Mirroristas/bigchaindb
2
0
Fork 0
mirror of https://github.com/bigchaindb/bigchaindb.git synced 2024-10-13 13:34:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

Secure WebSocket Support (#1619)

Browse Source 
* Remove support for whitelist

* Rename nginx to nginx-api

* Remove websocket support from nginx-api

* Change nginx to nginx-api service
The nginx-api service will proxy requests to the BigchainDB HTTP API.

* Rename ngx-instance-0 to ngx-api-instance-0 in nginx_3scale

* Update nginx-api service base docker image and README

* Add nginx-ws service to support Websocket

* Add config files for simple dev setup

* WS support with split NGINX

* NGINX module as single entrypoint into the cluster.

* Tested HTTP and WS with latest configs

* Openresty as separate service

* Remove upstream api port as configuration parameter

* Changes while testing

* Adding READMEs for nginx-http and nginx-https modules

* Documentation update

* Change 'Openresty' to 'OpenResty'.
This commit is contained in:
Krish 2017-07-17 13:46:05 +02:00 committed by GitHub
parent f576894b4c
commit b7e13fd087
35 changed files with 1634 additions and 630 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
docs/server/source/production-deployment-template/add-node-on-kubernetes.rst
View File

@ -71,10 +71,10 @@ Step 2: Prepare the New Kubernetes Cluster
Follow the steps in the sections to set up Storage Classes and Persistent Volume
Claims, and to run MongoDB in the new cluster:
1. :ref:`Add Storage Classes <Step 9: Create Kubernetes Storage Classes for MongoDB>`.
2. :ref:`Add Persistent Volume Claims <Step 10: Create Kubernetes Persistent Volume Claims>`.
1. :ref:`Add Storage Classes <Step 10: Create Kubernetes Storage Classes for MongoDB>`.
2. :ref:`Add Persistent Volume Claims <Step 11: Create Kubernetes Persistent Volume Claims>`.
3. :ref:`Create the Config Map <Step 3: Configure Your BigchainDB Node>`.
4. :ref:`Run MongoDB instance <Step 11: Start a Kubernetes StatefulSet for MongoDB>`.
4. :ref:`Run MongoDB instance <Step 12: Start a Kubernetes StatefulSet for MongoDB>`.
Step 3: Add the New MongoDB Instance to the Existing Replica Set
@ -166,13 +166,13 @@ show-config`` command to check that the keyring is updated.
Step 7: Run NGINX as a Deployment
---------------------------------
Please see :ref:`this page <Step 8: Start the NGINX Kubernetes Deployment>` to
Please see :ref:`this page <Step 9: Start the NGINX Kubernetes Deployment>` to
set up NGINX in your new node.
Step 8: Test Your New BigchainDB Node
-------------------------------------
Please refer to the testing steps :ref:`here <Step 17: Verify the BigchainDB
Please refer to the testing steps :ref:`here <Step 19: Verify the BigchainDB
Node Setup>` to verify that your new BigchainDB node is working as expected.

10
docs/server/source/production-deployment-template/index.rst
View File

@ -20,9 +20,9 @@ Feel free change things to suit your needs or preferences.
revoke-tls-certificate
template-kubernetes-azure
node-on-kubernetes
add-node-on-kubernetes
upgrade-on-kubernetes
log-analytics
easy-rsa
cloud-manager
node-config-map-and-secrets
log-analytics
cloud-manager
easy-rsa
upgrade-on-kubernetes
add-node-on-kubernetes

94
docs/server/source/production-deployment-template/node-config-map-and-secrets.rst
View File

@ -29,6 +29,39 @@ where all data values must be base64-encoded.
This is true of all Kubernetes ConfigMaps and Secrets.)
vars.cluster-fqdn
~~~~~~~~~~~~~~~~~
The ``cluster-fqdn`` field specifies the domain you would have
:ref:`registered before <2. Register a Domain and Get an SSL Certificate for It>`.
vars.cluster-frontend-port
~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``cluster-frontend-port`` field specifies the port on which your cluster
will be available to all external clients.
It is set to the HTTPS port ``443`` by default.
vars.cluster-health-check-port
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``cluster-healthcheck-port`` is the port number on which health check
probes are sent to the main NGINX instance.
It is set to ``8888`` by default.
vars.cluster-dns-server-ip
~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``cluster-dns-server-ip`` is the IP of the DNS server for a node.
We use DNS for service discovery. A Kubernetes deployment always has a DNS
server (``kube-dns``) running at 10.0.0.10, and since we use Kubernetes, this is
set to ``10.0.0.10`` by default, which is the default ``kube-dns`` IP address.
vars.mdb-instance-name and Similar
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -49,6 +82,57 @@ There are some things worth noting about the ``mdb-instance-name``:
* We use ``mdb-instance-0``, ``mdb-instance-1`` and so on in our
documentation. Your BigchainDB cluster may use a different naming convention.
vars.ngx-ndb-instance-name and Similar
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NGINX needs the FQDN of the servers inside the cluster to be able to forward
traffic.
The ``ngx-openresty-instance-name``, ``ngx-mdb-instance-name`` and
``ngx-bdb-instance-name`` are the FQDNs of the OpenResty instance, the MongoDB
instance, and the BigchainDB instance in this Kubernetes cluster respectively.
In Kubernetes, this is usually the name of the module specified in the
corresponding ``vars.*-instance-name`` followed by the
``<namespace name>.svc.cluster.local``. For example, if you run OpenResty in
the default Kubernetes namespace, this will be
``<vars.openresty-instance-name>.default.svc.cluster.local``
vars.mongodb-frontend-port and vars.mongodb-backend-port
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``mongodb-frontend-port`` is the port number on which external clients can
access MongoDB. This needs to be restricted to only other MongoDB instances
by enabling an authentication mechanism on MongoDB cluster.
It is set to ``27017`` by default.
The ``mongodb-backend-port`` is the port number on which MongoDB is actually
available/listening for requests in your cluster.
It is also set to ``27017`` by default.
vars.openresty-backend-port
~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``openresty-backend-port`` is the port number on which OpenResty is
listening for requests.
This is used by the NGINX instance to forward requests
destined for the OpenResty instance to the right port.
This is also used by OpenResty instance to bind to the correct port to
receive requests from NGINX instance.
It is set to ``80`` by default.
vars.bigchaindb-api-port and vars.bigchaindb-ws-port
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``bigchaindb-api-port`` is the port number on which BigchainDB is
listening for HTTP requests. Currently set to ``9984`` by default.
The ``bigchaindb-ws-port`` is the port number on which BigchainDB is
listening for Websocket requests. Currently set to ``9985`` by default.
bdb-config.bdb-keyring
~~~~~~~~~~~~~~~~~~~~~~~
@ -127,12 +211,10 @@ If you're not using 3scale,
you can delete the ``threescale-credentials`` Secret
or leave all the values blank (``""``).
If you *are* using 3scale, you can get the value for ``frontend-api-dns-name``
using something like ``echo "your.nodesubdomain.net" | base64 -w 0``
To get the values for ``secret-token``, ``service-id``,
``version-header`` and ``provider-key``, login to your 3scale admin,
then click **APIs** and click on **Integration** for the relevant API.
If you *are* using 3scale, get the values for ``secret-token``,
``service-id``, ``version-header`` and ``provider-key`` by logging in to 3scale
portal using your admin account, click **APIs** and click on **Integration**
for the relevant API.
Scroll to the bottom of the page and click the small link
in the lower right corner, labelled **Download the NGINX Config files**.
You'll get a ``.zip`` file.

279
docs/server/source/production-deployment-template/node-on-kubernetes.rst
View File

@ -113,46 +113,40 @@ Step 4: Start the NGINX Service
public IP to be assigned.
* You have the option to use vanilla NGINX without HTTPS support or an
OpenResty NGINX integrated with 3scale API Gateway.
NGINX with HTTPS support integrated with 3scale API Gateway.
Step 4.1: Vanilla NGINX
^^^^^^^^^^^^^^^^^^^^^^^
* This configuration is located in the file ``nginx/nginx-svc.yaml``.
* This configuration is located in the file ``nginx-http/nginx-http-svc.yaml``.
* Set the ``metadata.name`` and ``metadata.labels.name`` to the value
set in ``ngx-instance-name`` in the ConfigMap above.
* Set the ``spec.selector.app`` to the value set in ``ngx-instance-name`` in
the ConfigMap followed by ``-dep``. For example, if the value set in the
``ngx-instance-name`` is ``ngx-instance-0``, set the
``spec.selector.app`` to ``ngx-instance-0-dep``.
``ngx-instance-name`` is ``ngx-http-instance-0``, set the
``spec.selector.app`` to ``ngx-http-instance-0-dep``.
* Set ``ngx-public-mdb-port.port`` to 27017, or the port number on which you
want to expose MongoDB service.
Set the ``ngx-public-mdb-port.targetPort`` to the port number on which the
Kubernetes MongoDB service will be present.
* Set ``ports[0].port`` and ``ports[0].targetPort`` to the value set in the
``cluster-frontend-port`` in the ConfigMap above. This is the
``public-cluster-port`` in the file which is the ingress in to the cluster.
* Set ``ngx-public-api-port.port`` to 80, or the port number on which you want to
expose BigchainDB API service.
Set the ``ngx-public-api-port.targetPort`` to the port number on which the
Kubernetes BigchainDB API service will present.
* Set ``ports[1].port`` and ``ports[1].targetPort`` to the value set in the
``cluster-health-check-port`` in the ConfigMap above. This is the
``public-health-check-port`` in the file which is the health check port.
Note: This will be removed in the future.
* Set ``ngx-public-ws-port.port`` to 81, or the port number on which you want to
expose BigchainDB Websocket service.
Set the ``ngx-public-ws-port.targetPort`` to the port number on which the
BigchainDB Websocket service will be present.
* Start the Kubernetes Service:
.. code:: bash
$ kubectl --context k8s-bdb-test-cluster-0 apply -f nginx/nginx-svc.yaml
$ kubectl --context k8s-bdb-test-cluster-0 apply -f nginx-http/nginx-http-svc.yaml
Step 4.2: OpenResty NGINX + 3scale
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Step 4.2: NGINX with HTTPS + 3scale
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* You have to enable HTTPS for this one and will need an HTTPS certificate
for your domain.
@ -160,42 +154,42 @@ Step 4.2: OpenResty NGINX + 3scale
* You should have already created the necessary Kubernetes Secrets in the previous
step (e.g. ``https-certs`` and ``threescale-credentials``).
* This configuration is located in the file ``nginx-3scale/nginx-3scale-svc.yaml``.
* This configuration is located in the file ``nginx-https/nginx-https-svc.yaml``.
* Set the ``metadata.name`` and ``metadata.labels.name`` to the value
set in ``ngx-instance-name`` in the ConfigMap above.
* Set the ``spec.selector.app`` to the value set in ``ngx-instance-name`` in
the ConfigMap followed by ``-dep``. For example, if the value set in the
``ngx-instance-name`` is ``ngx-instance-0``, set the
``spec.selector.app`` to ``ngx-instance-0-dep``.
``ngx-instance-name`` is ``ngx-https-instance-0``, set the
``spec.selector.app`` to ``ngx-https-instance-0-dep``.
* Set ``ngx-public-mdb-port.port`` to 27017, or the port number on which you
* Set ``ports[0].port`` and ``ports[0].targetPort`` to the value set in the
``cluster-frontend-port`` in the ConfigMap above. This is the
``public-secure-cluster-port`` in the file which is the ingress in to the cluster.
* Set ``ports[1].port`` and ``ports[1].targetPort`` to the value set in the
``mongodb-frontend-port`` in the ConfigMap above. This is the
``public-mdb-port`` in the file which specifies where MongoDB is
available.
* Set ``ports[2].port`` and ``ports[2].targetPort`` to the value set in the
``threescale-api-port`` in the ConfigMap above. This is the
``public-threescale-port`` in the file which specifies where OpenResty is
available.
* Set ``threescale-api-port`` to 27017, or the port number on which you
want to expose MongoDB service.
Set the ``ngx-public-mdb-port.targetPort`` to the port number on which the
Kubernetes MongoDB service will be present.
Note: This is only used for testing with 3scale, and will be removed in
the future.
* Set ``ngx-public-3scale-port.port`` to 8080, or the port number on which
you want to let 3scale communicate with Openresty NGINX for authenctication.
Set the ``ngx-public-3scale-port.targetPort`` to the port number on which
this Openresty NGINX service will be listening to for communication with
3scale.
* Set ``ngx-public-bdb-port.port`` to 443, or the port number on which you want
to expose BigchainDB API service.
Set the ``ngx-public-api-port.targetPort`` to the port number on which the
Kubernetes BigchainDB API service will present.
* Set ``ngx-public-bdb-port-http.port`` to 80, or the port number on which you
want to expose BigchainDB Websocket service.
Set the ``ngx-public-bdb-port-http.targetPort`` to the port number on which the
BigchainDB Websocket service will be present.
* Start the Kubernetes Service:
.. code:: bash
$ kubectl --context k8s-bdb-test-cluster-0 apply -f nginx-3scale/nginx-3scale-svc.yaml
$ kubectl --context k8s-bdb-test-cluster-0 apply -f nginx-https/nginx-https-svc.yaml
Step 5: Assign DNS Name to the NGINX Public IP
@ -246,7 +240,7 @@ Step 6: Start the MongoDB Kubernetes Service
* Set the ``metadata.name`` and ``metadata.labels.name`` to the value
set in ``mdb-instance-name`` in the ConfigMap above.
* Set the ``spec.selector.app`` to the value set in ``mdb-instance-name`` in
the ConfigMap followed by ``-ss``. For example, if the value set in the
``mdb-instance-name`` is ``mdb-instance-0``, set the
@ -266,7 +260,7 @@ Step 7: Start the BigchainDB Kubernetes Service
* Set the ``metadata.name`` and ``metadata.labels.name`` to the value
set in ``bdb-instance-name`` in the ConfigMap above.
* Set the ``spec.selector.app`` to the value set in ``bdb-instance-name`` in
the ConfigMap followed by ``-dep``. For example, if the value set in the
``bdb-instance-name`` is ``bdb-instance-0``, set the
@ -279,72 +273,77 @@ Step 7: Start the BigchainDB Kubernetes Service
$ kubectl --context k8s-bdb-test-cluster-0 apply -f bigchaindb/bigchaindb-svc.yaml
Step 8: Start the NGINX Kubernetes Deployment
Step 8: Start the OpenResty Kubernetes Service
----------------------------------------------
* This configuration is located in the file ``nginx-openresty/nginx-openresty-svc.yaml``.
* Set the ``metadata.name`` and ``metadata.labels.name`` to the value
set in ``openresty-instance-name`` in the ConfigMap above.
* Set the ``spec.selector.app`` to the value set in ``openresty-instance-name`` in
the ConfigMap followed by ``-dep``. For example, if the value set in the
``openresty-instance-name`` is ``openresty-instance-0``, set the
``spec.selector.app`` to ``openresty-instance-0-dep``.
* Start the Kubernetes Service:
.. code:: bash
$ kubectl --context k8s-bdb-test-cluster-0 apply -f nginx-openresty/nginx-openresty-svc.yaml
Step 9: Start the NGINX Kubernetes Deployment
---------------------------------------------
* NGINX is used as a proxy to both the BigchainDB and MongoDB instances in
the node. It proxies HTTP requests on port 80 to the BigchainDB backend,
and TCP connections on port 27017 to the MongoDB backend.
* NGINX is used as a proxy to OpenResty, BigchainDB and MongoDB instances in
the node. It proxies HTTP/HTTPS requests on the ``clusted-frontend-port``
to the corresponding OpenResty or BigchainDB backend, and TCP connections
on ``mongodb-frontend-port`` to the MongoDB backend.
* As in step 4, you have the option to use vanilla NGINX or an OpenResty
NGINX integrated with 3scale API Gateway.
* As in step 4, you have the option to use vanilla NGINX without HTTPS or
NGINX with HTTPS support integrated with 3scale API Gateway.
Step 8.1: Vanilla NGINX
Step 9.1: Vanilla NGINX
^^^^^^^^^^^^^^^^^^^^^^^
* This configuration is located in the file ``nginx/nginx-dep.yaml``.
* This configuration is located in the file ``nginx-http/nginx-http-dep.yaml``.
* Set the ``metadata.name`` and ``spec.template.metadata.labels.app``
to the value set in ``ngx-instance-name`` in the ConfigMap followed by a
``-dep``. For example, if the value set in the ``ngx-instance-name`` is
``ngx-instance-0``, set the fields to ``ngx-instance-0-dep``.
``ngx-http-instance-0``, set the fields to ``ngx-http-instance-0-dep``.
* Set the ports to be exposed from the pod in the
``spec.containers[0].ports`` section. We currently expose 3 ports -
``mongodb-frontend-port``, ``cluster-frontend-port`` and
``cluster-health-check-port``. Set them to the values specified in the
ConfigMap.
* Set ``MONGODB_BACKEND_HOST`` env var to
the value set in ``mdb-instance-name`` in the ConfigMap, followed by
``.default.svc.cluster.local``. For example, if the value set in the
``mdb-instance-name`` is ``mdb-instance-0``, set the
``MONGODB_BACKEND_HOST`` env var to
``mdb-instance-0.default.svc.cluster.local``.
* Set ``BIGCHAINDB_BACKEND_HOST`` env var to
the value set in ``bdb-instance-name`` in the ConfigMap, followed by
``.default.svc.cluster.local``. For example, if the value set in the
``bdb-instance-name`` is ``bdb-instance-0``, set the
``BIGCHAINDB_BACKEND_HOST`` env var to
``bdb-instance-0.default.svc.cluster.local``.
* Start the Kubernetes Deployment:
.. code:: bash
$ kubectl --context k8s-bdb-test-cluster-0 apply -f nginx/nginx-dep.yaml
$ kubectl --context k8s-bdb-test-cluster-0 apply -f nginx-http/nginx-http-dep.yaml
Step 8.2: OpenResty NGINX + 3scale
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Step 9.2: NGINX with HTTPS + 3scale
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* This configuration is located in the file
``nginx-3scale/nginx-3scale-dep.yaml``.
``nginx-https/nginx-https-dep.yaml``.
* Set the ``metadata.name`` and ``spec.template.metadata.labels.app``
to the value set in ``ngx-instance-name`` in the ConfigMap followed by a
``-dep``. For example, if the value set in the ``ngx-instance-name`` is
``ngx-instance-0``, set the fields to ``ngx-instance-0-dep``.
``ngx-https-instance-0``, set the fields to ``ngx-https-instance-0-dep``.
* Set the ports to be exposed from the pod in the
``spec.containers[0].ports`` section. We currently expose 4 ports -
``mongodb-frontend-port``, ``cluster-frontend-port``,
``threescale-api-port`` and ``cluster-health-check-port``. Set them to
the values specified in the ConfigMap.
* Set ``MONGODB_BACKEND_HOST`` env var to
the value set in ``mdb-instance-name`` in the ConfigMap, followed by
``.default.svc.cluster.local``. For example, if the value set in the
``mdb-instance-name`` is ``mdb-instance-0``, set the
``MONGODB_BACKEND_HOST`` env var to
``mdb-instance-0.default.svc.cluster.local``.
* Set ``BIGCHAINDB_BACKEND_HOST`` env var to
the value set in ``bdb-instance-name`` in the ConfigMap, followed by
``.default.svc.cluster.local``. For example, if the value set in the
``bdb-instance-name`` is ``bdb-instance-0``, set the
``BIGCHAINDB_BACKEND_HOST`` env var to
``bdb-instance-0.default.svc.cluster.local``.
* Start the Kubernetes Deployment:
.. code:: bash
@ -352,8 +351,8 @@ Step 8.2: OpenResty NGINX + 3scale
$ kubectl --context k8s-bdb-test-cluster-0 apply -f nginx-3scale/nginx-3scale-dep.yaml
Step 9: Create Kubernetes Storage Classes for MongoDB
-----------------------------------------------------
Step 10: Create Kubernetes Storage Classes for MongoDB
------------------------------------------------------
MongoDB needs somewhere to store its data persistently,
outside the container where MongoDB is running.
@ -425,7 +424,7 @@ Kubernetes just looks for a storageAccount
with the specified skuName and location.
Step 10: Create Kubernetes Persistent Volume Claims
Step 11: Create Kubernetes Persistent Volume Claims
---------------------------------------------------
Next, you will create two PersistentVolumeClaim objects ``mongo-db-claim`` and
@ -457,7 +456,7 @@ Initially, the status of persistent volume claims might be "Pending"
but it should become "Bound" fairly quickly.
Step 11: Start a Kubernetes StatefulSet for MongoDB
Step 12: Start a Kubernetes StatefulSet for MongoDB
---------------------------------------------------
* This configuration is located in the file ``mongodb/mongo-ss.yaml``.
@ -512,7 +511,7 @@ Step 11: Start a Kubernetes StatefulSet for MongoDB
$ kubectl --context k8s-bdb-test-cluster-0 get pods -w
Step 12: Configure Users and Access Control for MongoDB
Step 13: Configure Users and Access Control for MongoDB
-------------------------------------------------------
* In this step, you will create a user on MongoDB with authorization
@ -640,7 +639,7 @@ Step 12: Configure Users and Access Control for MongoDB
} )
Step 13: Start a Kubernetes Deployment for MongoDB Monitoring Agent
Step 14: Start a Kubernetes Deployment for MongoDB Monitoring Agent
-------------------------------------------------------------------
* This configuration is located in the file
@ -661,7 +660,7 @@ Step 13: Start a Kubernetes Deployment for MongoDB Monitoring Agent
$ kubectl --context k8s-bdb-test-cluster-0 apply -f mongodb-monitoring-agent/mongo-mon-dep.yaml
Step 14: Start a Kubernetes Deployment for MongoDB Backup Agent
Step 15: Start a Kubernetes Deployment for MongoDB Backup Agent
---------------------------------------------------------------
* This configuration is located in the file
@ -682,7 +681,7 @@ Step 14: Start a Kubernetes Deployment for MongoDB Backup Agent
$ kubectl --context k8s-bdb-test-cluster-0 apply -f mongodb-backup-agent/mongo-backup-dep.yaml
Step 15: Start a Kubernetes Deployment for BigchainDB
Step 16: Start a Kubernetes Deployment for BigchainDB
-----------------------------------------------------
* This configuration is located in the file
@ -717,7 +716,35 @@ Step 15: Start a Kubernetes Deployment for BigchainDB
* You can check its status using the command ``kubectl get deployments -w``
Step 16: Configure the MongoDB Cloud Manager
Step 17: Start a Kubernetes Deployment for OpenResty
----------------------------------------------------
* This configuration is located in the file
``nginx-openresty/nginx-openresty-dep.yaml``.
* Set ``metadata.name`` and ``spec.template.metadata.labels.app`` to the
value set in ``openresty-instance-name`` in the ConfigMap, followed by
``-dep``.
For example, if the value set in the
``openresty-instance-name`` is ``openresty-instance-0``, set the fields to
the value ``openresty-instance-0-dep``.
* Set the port to be exposed from the pod in the
``spec.containers[0].ports`` section. We currently expose the port at
which OpenResty is listening for requests, ``openresty-backend-port`` in
the above ConfigMap.
* Create the OpenResty Deployment using:
.. code:: bash
$ kubectl --context k8s-bdb-test-cluster-0 apply -f nginx-openresty/nginx-openresty-dep.yaml
* You can check its status using the command ``kubectl get deployments -w``
Step 18: Configure the MongoDB Cloud Manager
--------------------------------------------
Refer to the
@ -726,10 +753,10 @@ for details on how to configure the MongoDB Cloud Manager to enable
monitoring and backup.
Step 17: Verify the BigchainDB Node Setup
Step 19: Verify the BigchainDB Node Setup
-----------------------------------------
Step 17.1: Testing Internally
Step 19.1: Testing Internally
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To test the setup of your BigchainDB node, you could use a Docker container
@ -773,38 +800,69 @@ To test the BigchainDB instance:
$ nslookup bdb-instance-0
$ dig +noall +answer _bdb-port._tcp.bdb-instance-0.default.svc.cluster.local SRV
$ dig +noall +answer _bdb-ws-port._tcp.bdb-instance-0.default.svc.cluster.local SRV
$ curl -X GET http://bdb-instance-0:9984
$ wsc -er ws://bdb-instance-0:9985/api/v1/streams/valid_transactions
To test the NGINX instance:
To test the OpenResty instance:
.. code:: bash
$ nslookup openresty-instance-0
$ dig +noall +answer _openresty-svc-port._tcp.openresty-instance-0.default.svc.cluster.local SRV
To verify if OpenResty instance forwards the requests properly, send a ``POST``
transaction to OpenResty at post ``80`` and check the response from the backend
BigchainDB instance.
To test the vanilla NGINX instance:
.. code:: bash
$ nslookup ngx-instance-0
$ nslookup ngx-http-instance-0
$ dig +noall +answer _ngx-public-mdb-port._tcp.ngx-instance-0.default.svc.cluster.local SRV
$ dig +noall +answer _public-cluster-port._tcp.ngx-http-instance-0.default.svc.cluster.local SRV
$ dig +noall +answer _ngx-public-bdb-port._tcp.ngx-instance-0.default.svc.cluster.local SRV
$ dig +noall +answer _public-health-check-port._tcp.ngx-http-instance-0.default.svc.cluster.local SRV
$ curl -X GET http://ngx-instance-0:27017
$ wsc -er ws://ngx-http-instance-0/api/v1/streams/valid_transactions
The curl command should result get the response
``curl: (7) Failed to connect to ngx-instance-0 port 27017: Connection refused``.
$ curl -X GET http://ngx-http-instance-0:27017
If you ran the vanilla NGINX instance, run:
The above curl command should result in the response
``It looks like you are trying to access MongoDB over HTTP on the native driver port.``
To test the NGINX instance with HTTPS and 3scale integration:
.. code:: bash
$ nslookup ngx-https-instance-0
$ curl -X GET http://ngx-instance-0:80
If you ran the OpenResty NGINX + 3scale instance, run:
$ dig +noall +answer _public-secure-cluster-port._.tcp.ngx-https-instance-0.default.svc.cluster.local SRV
.. code:: bash
$ dig +noall +answer _public-mdb-port._.tcp.ngx-https-instance-0.default.svc.cluster.local SRV
$ curl -X GET https://ngx-instance-0
$ dig +noall +answer _public-threescale-port._.tcp.ngx-https-instance-0.default.svc.cluster.local SRV
$ dig +noall +answer _public-insecure-cluster-port._.tcp.ngx-https-instance-0.default.svc.cluster.local SRV
$ wsc -er wss://ngx-https-instance-0/api/v1/streams/valid_transactions
$ curl -X GET http://ngx-https-instance-0:27017
The above curl command should result in the response
``It looks like you are trying to access MongoDB over HTTP on the native driver port.``
Step 17.2: Testing Externally
Step 19.2: Testing Externally
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Check the MongoDB monitoring and backup agent on the MongoDB Cloud Manager
@ -816,3 +874,4 @@ server version, among other things.
Use the Python Driver to send some transactions to the BigchainDB node and
verify that your node or cluster works as expected.

3
docs/server/source/production-deployment-template/workflow.rst
View File

@ -48,7 +48,8 @@ Similarly, other instances must also have unique names in the cluster.
#. Name of the MongoDB instance (``mdb-instance-*``)
#. Name of the BigchainDB instance (``bdb-instance-*``)
#. Name of the NGINX instance (``ngx-instance-*``)
#. Name of the NGINX instance (``ngx-http-instance-*`` or ``ngx-https-instance-*``)
#. Name of the OpenResty instance (``openresty-instance-*``)
#. Name of the MongoDB monitoring agent instance (``mdb-mon-instance-*``)
#. Name of the MongoDB backup agent instance (``mdb-bak-instance-*``)

88
k8s/configuration/config-map.yaml
View File

@ -7,25 +7,81 @@ metadata:
name: vars
namespace: default
data:
# MongoDB
# cluster-fqdn is the DNS name registered for your HTTPS certificate.
cluster-fqdn: "bdb.example.com"
# cluster-frontend-port is the port number on which this node's services
# are available to external clients.
cluster-frontend-port: "443"
# cluster-health-check-port is the port number on which an external load
# balancer can check the status/liveness of the external/public server.
cluster-health-check-port: "8888"
# cluster-dns-server-ip is the IP of the DNS server. A Kubernetes deployment
# always has a DNS server (kube-dns) running at 10.0.0.10
cluster-dns-server-ip: "10.0.0.10"
# mdb-instance-name is the name of the MongoDB instance in this cluster.
mdb-instance-name: "<name of the mdb instance>"
# BigchainDB
# ngx-instance-name is the name of the NGINX instance in this cluster.
ngx-instance-name: "<name of the nginx instance>"
# openresty-instance-name is the name of the OpenResty instance in this
# cluster.
openresty-instance-name: "<name of the openresty instance>"
# bdb-instance-name is the name of the BigchainDB instance in this cluster.
bdb-instance-name: "<name of the bdb instance>"
# NGINX
ngx-instance-name: "<name of the ngx instance>"
# MongoDB Monitoring Agent
# mdb-mon-instance-name is the name of the MongoDB Monitoring Agent instance
# in this cluster.
mdb-mon-instance-name: "<name of the mdb monitoring agent instance>"
# MongoDB Backup Agent
# mdb-bak-instance-name is the name of the MongoDB Backup Agent instance
# in this cluster.
mdb-bak-instance-name: "<name of the mdb backup agent instance>"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: mongodb-whitelist
namespace: default
data:
# We only support "all"" currently
allowed-hosts: "all"
# ngx-mdb-instance-name is the FQDN of the MongoDB instance in this
# Kubernetes cluster.
ngx-mdb-instance-name: "<name of the mdb instance>.default.svc.cluster.local"
# ngx-openresty-instance-name is the FQDN of the MongoDB instance in this
# Kubernetes cluster.
ngx-openresty-instance-name: "<name of the openresty instance>.default.svc.cluster.local"
# ngx-bdb-instance-name is the FQDN of the BigchainDB instance in this
# Kubernetes cluster.
ngx-bdb-instance-name: "<name of the bdb instance>.default.svc.cluster.local"
# mongodb-frontend-port is the port number on which external clients can
# access MongoDB. This needs to be restricted to only other MongoDB instances
# by enabling an authentication mechanism on MongoDB.
mongodb-frontend-port: "27017"
# mongodb-backend-port is the port on which MongoDB is actually
# available/listening for requests.
mongodb-backend-port: "27017"
# openresty-backend-port is the port number on which OpenResty is listening
# for requests. This is used by the NGINX instance to forward the requests to
# the right port, and by OpenResty instance to bind to the correct port to
# receive requests from NGINX instance.
openresty-backend-port: "80"
# bigchaindb-api-port is the port number on which BigchainDB is listening
# for HTTP requests.
bigchaindb-api-port: "9984"
# bigchaindb-ws-port is the port number on which BigchainDB is listening
# for Websocket requests.
bigchaindb-ws-port: "9985"
# threescale-api-port is the port number on which 3scale can communicate with
# OpenResty backend.
threescale-api-port: "44433"
---
apiVersion: v1
kind: ConfigMap
@ -35,8 +91,10 @@ metadata:
data:
# Colon-separated list of all *other* nodes' BigchainDB public keys.
bdb-keyring: "<':' separated list of public keys>"
# BigchainDB instance authentication user name
bdb-user: "<user name>"
# BigchainDB public key of *this* node.
# Example: "EPQk5i5yYpoUwGVM8VKZRjM8CYxB6j8Lu8i8SG7kGGce"
bdb-public-key: "<public key>"

6
k8s/configuration/secret.yaml
View File

@ -109,9 +109,3 @@ data:
service-id: "<b64 encoded 3scale service-id>"
version-header: "<b64 encoded 3scale version-header>"
provider-key: "<b64 encoded 3scale provider-key>"
# The frontend-api-dns-name will be DNS name registered for your HTTPS
# certificate.
frontend-api-dns-name: "<b64 encoded DNS/FQDN>"
# The upstream-api-port can be set to any port other than 9984, 9985, 443,
# 8888 and 27017. We usually use port '9999', which is 'OTk5OQo=' in base 64.
upstream-api-port: "OTk5OQo="

95
k8s/dev-setup/bigchaindb.yaml Normal file
View File

@ -0,0 +1,95 @@
apiVersion: v1
kind: Service
metadata:
name: bdb
namespace: default
labels:
name: bdb
spec:
selector:
app: bdb-dep
ports:
- port: 9984
targetPort: 9984
name: bdb-api-port
protocol: TCP
- port: 9985
targetPort: 9985
name: bdb-ws-port
protocol: TCP
type: ClusterIP
clusterIP: None
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: bdb-dep
spec:
replicas: 1
template:
metadata:
labels:
app: bdb-dep
spec:
terminationGracePeriodSeconds: 10
containers:
- name: bigchaindb
image: bigchaindb/bigchaindb:1.0.0
imagePullPolicy: Always
args:
- start
env:
- name: BIGCHAINDB_DATABASE_HOST
value: mdb
- name: BIGCHAINDB_DATABASE_PORT
value: "27017"
- name: BIGCHAINDB_DATABASE_REPLICASET
value: bigchain-rs
- name: BIGCHAINDB_DATABASE_BACKEND
value: mongodb
- name: BIGCHAINDB_DATABASE_NAME
value: bigchain
- name: BIGCHAINDB_SERVER_BIND
value: "0.0.0.0:9984"
- name: BIGCHAINDB_WSSERVER_HOST
value: "0.0.0.0"
- name: BIGCHAINDB_WSSERVER_PORT
value: "9985"
- name: BIGCHAINDB_KEYPAIR_PUBLIC
value: "EEWUAhsk94ZUHhVw7qx9oZiXYDAWc9cRz93eMrsTG4kZ"
- name: BIGCHAINDB_KEYPAIR_PRIVATE
value: "3CjmRhu718gT1Wkba3LfdqX5pfYuBdaMPLd7ENUga5dm"
- name: BIGCHAINDB_BACKLOG_REASSIGN_DELAY
value: "120"
- name: BIGCHAINDB_DATABASE_MAXTRIES
value: "3"
- name: BIGCHAINDB_DATABASE_CONNECTION_TIMEOUT
value: "120"
- name: BIGCHAINDB_LOG_LEVEL_CONSOLE
value: debug
ports:
- containerPort: 9984
hostPort: 9984
name: bdb-port
protocol: TCP
- containerPort: 9985
hostPort: 9985
name: bdb-ws-port
protocol: TCP
resources:
limits:
cpu: 200m
memory: 768Mi
livenessProbe:
httpGet:
path: /
port: 9984
initialDelaySeconds: 15
timeoutSeconds: 10
readinessProbe:
httpGet:
path: /
port: 9984
initialDelaySeconds: 15
timeoutSeconds: 10
restartPolicy: Always

49
k8s/dev-setup/mongo.yaml Normal file
View File

@ -0,0 +1,49 @@
apiVersion: v1
kind: Service
metadata:
name: mdb
namespace: default
labels:
name: mdb
spec:
selector:
app: mdb-ss
ports:
- port: 27017
targetPort: 27017
protocol: TCP
name: mdb-svc-port
type: ClusterIP
clusterIP: None
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: mdb-ss
namespace: default
spec:
serviceName: mdb
replicas: 1
template:
metadata:
name: mdb-ss
labels:
app: mdb-ss
spec:
terminationGracePeriodSeconds: 10
containers:
- name: mongodb
image: mongo:3.4.4
imagePullPolicy: Always
args:
- --replSet
- bigchain-rs
ports:
- containerPort: 27017
hostPort: 27017
protocol: TCP
resources:
limits:
cpu: 200m
memory: 768Mi
restartPolicy: Always

79
k8s/dev-setup/nginx-http.yaml Normal file
View File

@ -0,0 +1,79 @@
apiVersion: v1
kind: Service
metadata:
name: ngx-http
namespace: default
labels:
name: ngx-http
annotations:
# NOTE: the following annotation is a beta feature and
# only available in GCE/GKE and Azure as of now
# Ref: https://kubernetes.io/docs/tutorials/services/source-ip/
service.beta.kubernetes.io/external-traffic: OnlyLocal
spec:
selector:
app: ngx-http-dep
ports:
- port: 80
targetPort: 80
name: ngx-public-bdb-port-http
protocol: TCP
- port: 8080
targetPort: 8080
name: public-health-check-port
protocol: TCP
- port: 27017
targetPort: 27017
name: ngx-public-mdb-port
protocol: TCP
type: LoadBalancer
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: ngx-http-dep
namespace: default
spec:
replicas: 1
template:
metadata:
name: ngx-http-dep
labels:
app: ngx-http-dep
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx-http
image: bigchaindb/nginx_http:1.0
imagePullPolicy: Always
env:
- name: CLUSTER_FRONTEND_PORT
value: "80"
- name: HEALTH_CHECK_PORT
value: "8080"
- name: DNS_SERVER
value: "10.0.0.10"
- name: MONGODB_FRONTEND_PORT
value: "27017"
- name: MONGODB_BACKEND_HOST
value: "mdb.default.svc.cluster.local"
- name: MONGODB_BACKEND_PORT
value: "27017"
- name: BIGCHAINDB_BACKEND_HOST
value: "bdb.default.svc.cluster.local"
- name: BIGCHAINDB_API_PORT
value: "9984"
- name: BIGCHAINDB_WS_PORT
value: "9985"
ports:
- containerPort: 27017
protocol: TCP
- containerPort: 8080
protocol: TCP
- containerPort: 80
protocol: TCP
resources:
limits:
cpu: 200m
memory: 768Mi
restartPolicy: Always

127
k8s/dev-setup/nginx-https.yaml Normal file
View File

@ -0,0 +1,127 @@
apiVersion: v1
kind: Secret
metadata:
name: https-certs
namespace: default
type: Opaque
data:
# Base64-encoded HTTPS private key
cert.key: "<b64 encoded HTTPS private key>"
# Base64-encoded HTTPS certificate chain
# starting with your primary SSL cert (e.g. your_domain.crt)
# followed by all intermediate certs.
# If cert if from DigiCert, download "Best format for nginx".
cert.pem: "<b64 encoded HTTPS certificate chain"
# Base64-encoded HTTPS private key
---
apiVersion: v1
kind: Service
metadata:
name: ngx-https
namespace: default
labels:
name: ngx-https
annotations:
service.beta.kubernetes.io/external-traffic: OnlyLocal
spec:
selector:
app: ngx-https-dep
ports:
- port: 443
targetPort: 443
name: public-secure-cluster-port
protocol: TCP
- port: 80
targetPort: 80
name: public-insecure-cluster-port
protocol: TCP
- port: 27017
targetPort: 27017
name: public-mdb-port
protocol: TCP
- port: 44433
targetPort: 44433
name: public-threescale-port
protocol: TCP
type: LoadBalancer
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: ngx-https-dep
spec:
replicas: 1
template:
metadata:
labels:
app: ngx-https-dep
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx-https
image: bigchaindb/nginx_https:1.0
imagePullPolicy: Always
env:
- name: CLUSTER_FRONTEND_PORT
value: "443"
- name: HEALTH_CHECK_PORT
value: "8888"
- name: CLUSTER_FQDN
value: "unfake.io"
- name: DNS_SERVER
value: "10.0.0.10"
- name: MONGODB_FRONTEND_PORT
value: "27017"
- name: MONGODB_BACKEND_HOST
value: "mdb.default.svc.cluster.local"
- name: MONGODB_BACKEND_PORT
value: "27017"
- name: OPENRESTY_BACKEND_PORT
value: "80"
- name: OPENRESTY_BACKEND_HOST
value: "openresty.default.svc.cluster.local"
- name: THREESCALE_API_PORT
value: "44433"
- name: BIGCHAINDB_BACKEND_HOST
value: "bdb.default.svc.cluster.local"
- name: BIGCHAINDB_API_PORT
value: "9984"
- name: BIGCHAINDB_WS_PORT
value: "9985"
ports:
- containerPort: 80
protocol: TCP
- containerPort: 27017
protocol: TCP
- containerPort: 8888
protocol: TCP
- containerPort: 443
protocol: TCP
- containerPort: 44433
protocol: TCP
resources:
limits:
cpu: 200m
memory: 768Mi
volumeMounts:
- name: https-certs
mountPath: /etc/nginx/ssl/
readOnly: true
resources:
limits:
cpu: 200m
memory: 768Mi
livenessProbe:
httpGet:
path: /health
port: 8888
initialDelaySeconds: 15
periodSeconds: 15
failureThreshold: 3
timeoutSeconds: 10
restartPolicy: Always
volumes:
- name: https-certs
secret:
secretName: https-certs
defaultMode: 0400

83
k8s/dev-setup/nginx-openresty.yaml Normal file
View File

@ -0,0 +1,83 @@
apiVersion: v1
kind: Secret
metadata:
name: threescale-credentials
namespace: default
type: Opaque
data:
secret-token: "<b64 encoded 3scale secret-token>"
service-id: "<b64 encoded 3scale service-id>"
version-header: "<b64 encoded 3scale version-header>"
provider-key: "<b64 encoded 3scale provider-key>"
---
apiVersion: v1
kind: Service
metadata:
name: openresty
namespace: default
labels:
name: openresty
annotations:
service.beta.kubernetes.io/external-traffic: OnlyLocal
spec:
selector:
app: openresty-dep
ports:
- port: 80
targetPort: 80
name: openresty-svc-port
protocol: TCP
type: ClusterIP
clusterIP: None
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: openresty-dep
spec:
replicas: 1
template:
metadata:
labels:
app: openresty-dep
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx-openresty
image: bigchaindb/nginx_3scale:2.0
imagePullPolicy: Always
env:
- name: DNS_SERVER
value: "10.0.0.10"
- name: OPENRESTY_FRONTEND_PORT
value: "80"
- name: BIGCHAINDB_BACKEND_HOST
value: "bdb.default.svc.cluster.local"
- name: BIGCHAINDB_API_PORT
value: "9984"
ports:
- containerPort: 80
protocol: TCP
name: openresty-port
volumeMounts:
- name: threescale-credentials
mountPath: /usr/local/openresty/nginx/conf/threescale
readOnly: true
livenessProbe:
httpGet:
path: /
port: openresty-port
initialDelaySeconds: 15
periodSeconds: 15
failureThreshold: 3
timeoutSeconds: 10
resources:
limits:
cpu: 200m
memory: 768Mi
restartPolicy: Always
volumes:
- name: threescale-credentials
secret:
secretName: threescale-credentials
defaultMode: 0400

94
k8s/nginx-3scale/nginx-3scale-dep.yaml
View File

@ -1,94 +0,0 @@
###############################################################
# This config file runs nginx as a k8s deployment and exposes #
# it using an external load balancer. #
# This deployment is used as a front end to both BigchainDB #
# and MongoDB. #
###############################################################
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: ngx-instance-0-dep
spec:
replicas: 1
template:
metadata:
labels:
app: ngx-instance-0-dep
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx-3scale
image: bigchaindb/nginx_3scale:1.5
imagePullPolicy: Always
env:
- name: MONGODB_FRONTEND_PORT
value: $(NGX_INSTANCE_0_SERVICE_PORT_NGX_PUBLIC_MDB_PORT)
- name: MONGODB_BACKEND_HOST
# NGINX requires FQDN to resolve names
value: mdb-instance-0.default.svc.cluster.local
- name: MONGODB_BACKEND_PORT
value: "27017"
- name: BIGCHAINDB_FRONTEND_PORT
value: $(NGX_INSTANCE_0_SERVICE_PORT_NGX_PUBLIC_BDB_PORT)
- name: BIGCHAINDB_BACKEND_HOST
value: bdb-instance-0.default.svc.cluster.local
- name: BIGCHAINDB_BACKEND_PORT
value: "9984"
- name: MONGODB_WHITELIST
valueFrom:
configMapKeyRef:
name: mongodb-whitelist
key: allowed-hosts
- name: DNS_SERVER
value: "10.0.0.10"
- name: NGINX_HEALTH_CHECK_PORT
value: "8888"
ports:
- containerPort: 27017
hostPort: 27017
name: public-mdb-port
protocol: TCP
- containerPort: 443
hostPort: 443
name: public-bdb-port
protocol: TCP
- containerPort: 80
hostPort: 80
name: https-msg-port
protocol: TCP
- containerPort: 8888
hostPort: 8888
name: health-check
protocol: TCP
- containerPort: 8080
hostPort: 8080
name: public-api-port
protocol: TCP
volumeMounts:
- name: threescale-credentials
mountPath: /usr/local/openresty/nginx/conf/threescale
readOnly: true
- name: https-certs
mountPath: /usr/local/openresty/nginx/conf/ssl/
readOnly: true
resources:
limits:
cpu: 200m
memory: 768Mi
livenessProbe:
httpGet:
path: /
port: 8888
initialDelaySeconds: 15
timeoutSeconds: 10
restartPolicy: Always
volumes:
- name: https-certs
secret:
secretName: https-certs
defaultMode: 0400
- name: threescale-credentials
secret:
secretName: threescale-credentials
defaultMode: 0400

33
k8s/nginx-3scale/nginx-3scale-svc.yaml
View File

@ -1,33 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: ngx-instance-0
namespace: default
labels:
name: ngx-instance-0
annotations:
# NOTE: the following annotation is a beta feature and
# only available in GCE/GKE and Azure as of now
# Ref: https://kubernetes.io/docs/tutorials/services/source-ip/
service.beta.kubernetes.io/external-traffic: OnlyLocal
spec:
selector:
app: ngx-instance-0-dep
ports:
- port: 80
targetPort: 80
name: ngx-public-bdb-port-http
protocol: TCP
- port: 443
targetPort: 443
name: ngx-public-bdb-port
protocol: TCP
- port: 8080
targetPort: 8080
name: ngx-public-3scale-port
protocol: TCP
- port: 27017
targetPort: 27017
name: ngx-public-mdb-port
protocol: TCP
type: LoadBalancer

4
k8s/nginx/container/Dockerfile → k8s/nginx-http/container/Dockerfile
View File

@ -1,4 +1,4 @@
FROM nginx:1.11.10
FROM nginx:1.13.1
LABEL maintainer "dev@bigchaindb.com"
WORKDIR /
RUN apt-get update \
@ -7,5 +7,5 @@ RUN apt-get update \
&& apt-get clean
COPY nginx.conf.template /etc/nginx/nginx.conf
COPY nginx_entrypoint.bash /
EXPOSE 80 81 443 444 27017
EXPOSE 80 27017
ENTRYPOINT ["/nginx_entrypoint.bash"]

15
k8s/nginx-http/container/README.md Normal file
View File

@ -0,0 +1,15 @@
## Nginx container for Secure WebSocket Support
### Step 1: Build and Push the Latest Container
Use the `docker_build_and_push.bash` script to build the latest docker image
and upload it to Docker Hub.
Ensure that the image tag is updated to a new version number to properly
reflect any changes made to the container.
### Note about testing Websocket connections:
You can test the WebSocket server by using
[wsc](https://www.npmjs.com/package/wsc) tool with a command like:
`wsc -er ws://localhost:9985/api/v1/streams/valid_transactions`.

5
k8s/nginx-http/container/docker_build_and_push.bash Executable file
View File

@ -0,0 +1,5 @@
#!/bin/bash
docker build -t bigchaindb/nginx_http:1.0 .
docker push bigchaindb/nginx_http:1.0

154
k8s/nginx-http/container/nginx.conf.template Normal file
View File

@ -0,0 +1,154 @@
# Frontend API server that:
# 1. Forwards BDB HTTP & WS requests to BDB backend.
# 2. Forwards MDB TCP connections to MDB backend.
# 3. Does health check with LB (optional).
worker_processes 2;
daemon off;
user nobody nogroup;
pid /tmp/nginx.pid;
error_log /dev/stderr;
events {
# Each worker handles up to 512 connections. Increase this for heavy
# workloads.
worker_connections 512;
accept_mutex on;
use epoll;
}
http {
access_log /dev/stdout combined buffer=16k flush=5s;
# Allow 10 req/sec from the same IP address, and store the counters in a
# `zone` or shared memory location tagged as 'one'.
limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;
# Enable logging when requests are being throttled.
limit_req_log_level notice;
# HTTP status code to return to the client when throttling;
# 429 is for TooManyRequests, ref. RFC 6585
limit_req_status 429;
# Limit requests from the same client, allow `burst` to 20 r/s,
# `nodelay` or drop connection immediately in case it exceeds this
# threshold.
limit_req zone=one burst=20 nodelay;
# `slowloris` attack mitigation settings.
client_body_timeout 10s;
client_header_timeout 10s;
# DNS resolver to use for all the backend names specified in this configuration.
resolver DNS_SERVER valid=30s ipv6=off;
keepalive_timeout 60s;
# The following map blocks enable lazy-binding to the backend at runtime,
# rather than binding as soon as NGINX starts.
map $remote_addr $bdb_backend {
default BIGCHAINDB_BACKEND_HOST;
}
# Frontend server for the external clients
server {
listen CLUSTER_FRONTEND_PORT;
underscores_in_headers on;
# Forward websockets to backend BDB at 9985.
location /api/v1/streams/valid_transactions {
proxy_pass http://$bdb_backend:BIGCHAINDB_WS_PORT;
proxy_read_timeout 600s;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
# Forward other URL paths to backend BDB at 9984.
location / {
proxy_ignore_client_abort on;
proxy_set_header X-Real-IP $remote_addr;
# max client request body size: avg transaction size.
client_max_body_size 15k;
# No auth for GETs, forward directly to BDB.
if ($request_method = GET) {
proxy_pass http://$bdb_backend:BIGCHAINDB_API_PORT;
}
# POST requests get forwarded to OpenResty instance. Enable CORS too.
if ($request_method = POST ) {
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
add_header 'Access-Control-Expose-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
proxy_pass http://$bdb_backend:BIGCHAINDB_API_PORT;
}
# OPTIONS requests handling for CORS.
if ($request_method = 'OPTIONS') {
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range,app_key,app_id';
add_header 'Access-Control-Max-Age' 43200;
add_header 'Content-Type' 'text/plain charset=UTF-8';
add_header 'Content-Length' 0;
return 204;
}
}
}
# # Frontend server for the load balancer to respond to health checks.
# server {
# listen HEALTH_CHECK_PORT;
#
# location = /health {
# return 200;
# }
# }
}
# NGINX stream block for TCP and UDP proxies. Used to proxy MDB TCP
# connection.
stream {
log_format mdb_log '[$time_iso8601] $realip_remote_addr $remote_addr '
'$proxy_protocol_addr $proxy_protocol_port '
'$protocol $status $session_time $bytes_sent '
'$bytes_received "$upstream_addr" "$upstream_bytes_sent" '
'"$upstream_bytes_received" "$upstream_connect_time" ';
access_log /dev/stdout mdb_log buffer=16k flush=5s;
# Define a zone 'two' of size 10 megabytes to store the counters
# that hold number of TCP connections from a specific IP address.
limit_conn_zone $binary_remote_addr zone=two:10m;
# Enable logging when connections are being throttled.
limit_conn_log_level notice;
# Allow 16 connections from the same IP address.
limit_conn two 16;
# DNS resolver to use for all the backend names specified in this configuration.
resolver DNS_SERVER valid=30s ipv6=off;
# The following map block enables lazy-binding to the backend at runtime,
# rather than binding as soon as NGINX starts.
map $remote_addr $mdb_backend {
default MONGODB_BACKEND_HOST;
}
# Frontend server to forward connections to MDB instance.
server {
listen MONGODB_FRONTEND_PORT so_keepalive=10m:1m:5;
preread_timeout 30s;
tcp_nodelay on;
proxy_pass $mdb_backend:MONGODB_BACKEND_PORT;
}
}

55
k8s/nginx-http/container/nginx_entrypoint.bash Executable file
View File

@ -0,0 +1,55 @@
#!/bin/bash
set -euo pipefail
# Cluster vars
cluster_frontend_port=`printenv CLUSTER_FRONTEND_PORT`
# NGINX vars
dns_server=`printenv DNS_SERVER`
health_check_port=`printenv HEALTH_CHECK_PORT`
# MongoDB vars
mongo_frontend_port=`printenv MONGODB_FRONTEND_PORT`
mongo_backend_host=`printenv MONGODB_BACKEND_HOST`
mongo_backend_port=`printenv MONGODB_BACKEND_PORT`
# BigchainDB vars
bdb_backend_host=`printenv BIGCHAINDB_BACKEND_HOST`
bdb_api_port=`printenv BIGCHAINDB_API_PORT`
bdb_ws_port=`printenv BIGCHAINDB_WS_PORT`
# sanity check
if [[ -z "${cluster_frontend_port}" || \
-z "${mongo_frontend_port}" || \
-z "${mongo_backend_host}" || \
-z "${mongo_backend_port}" || \
-z "${bdb_backend_host}" || \
-z "${bdb_api_port}" || \
-z "${bdb_ws_port}" || \
-z "${dns_server}" || \
-z "${health_check_port}" ]]; then
echo "Invalid environment settings detected. Exiting!"
exit 1
fi
NGINX_CONF_FILE=/etc/nginx/nginx.conf
# configure the nginx.conf file with env variables
sed -i "s|CLUSTER_FRONTEND_PORT|${cluster_frontend_port}|g" ${NGINX_CONF_FILE}
sed -i "s|MONGODB_FRONTEND_PORT|${mongo_frontend_port}|g" ${NGINX_CONF_FILE}
sed -i "s|MONGODB_BACKEND_HOST|${mongo_backend_host}|g" ${NGINX_CONF_FILE}
sed -i "s|MONGODB_BACKEND_PORT|${mongo_backend_port}|g" ${NGINX_CONF_FILE}
sed -i "s|BIGCHAINDB_BACKEND_HOST|${bdb_backend_host}|g" ${NGINX_CONF_FILE}
sed -i "s|BIGCHAINDB_API_PORT|${bdb_api_port}|g" ${NGINX_CONF_FILE}
sed -i "s|BIGCHAINDB_WS_PORT|${bdb_ws_port}|g" ${NGINX_CONF_FILE}
sed -i "s|DNS_SERVER|${dns_server}|g" ${NGINX_CONF_FILE}
sed -i "s|HEALTH_CHECK_PORT|${health_check_port}|g" ${NGINX_CONF_FILE}
# start nginx
echo "INFO: starting nginx..."
exec nginx -c /etc/nginx/nginx.conf

74
k8s/nginx-http/nginx-http-dep.yaml Normal file
View File

@ -0,0 +1,74 @@
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: ngx-http-instance-0-dep
spec:
replicas: 1
template:
metadata:
labels:
app: ngx-http-instance-0-dep
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx-http
image: bigchaindb/nginx_http:1.0
imagePullPolicy: Always
env:
- name: CLUSTER_FRONTEND_PORT
valueFrom:
configMapKeyRef:
name: vars
key: cluster-frontend-port
- name: HEALTH_CHECK_PORT
valueFrom:
configMapKeyRef:
name: vars
key: cluster-health-check-port
- name: DNS_SERVER
valueFrom:
configMapKeyRef:
name: vars
key: cluster-dns-server-ip
- name: MONGODB_FRONTEND_PORT
valueFrom:
configMapKeyRef:
name: vars
key: mongodb-frontend-port
- name: MONGODB_BACKEND_HOST
valueFrom:
configMapKeyRef:
name: vars
key: ngx-mdb-instance-name
- name: MONGODB_BACKEND_PORT
valueFrom:
configMapKeyRef:
name: vars
key: mongodb-backend-port
- name: BIGCHAINDB_BACKEND_HOST
valueFrom:
configMapKeyRef:
name: vars
key: ngx-bdb-instance-name
- name: BIGCHAINDB_API_PORT
valueFrom:
configMapKeyRef:
name: vars
key: bigchaindb-api-port
- name: BIGCHAINDB_WS_PORT
valueFrom:
configMapKeyRef:
name: vars
key: bigchaindb-ws-port
ports:
- containerPort: "<mongodb-frontend-port from ConfigMap>"
protocol: TCP
- containerPort: "<cluster-health-check-port from ConfigMap>"
protocol: TCP
- containerPort: "<cluster-frontend-port from ConfigMap>"
protocol: TCP
resources:
limits:
cpu: 200m
memory: 768Mi
restartPolicy: Always

24
k8s/nginx-http/nginx-http-svc.yaml Normal file
View File

@ -0,0 +1,24 @@
apiVersion: v1
kind: Service
metadata:
name: ngx-http-instance-0
namespace: default
labels:
name: ngx-http-instance-0
annotations:
# NOTE: the following annotation is a beta feature and
# only available in GCE/GKE and Azure as of now
service.beta.kubernetes.io/external-traffic: OnlyLocal
spec:
selector:
app: ngx-http-instance-0-dep
ports:
- port: "<cluster-frontend-port from ConfigMap>"
targetPort: "<cluster-frontend-port from ConfigMap>"
name: public-cluster-port
protocol: TCP
- port: "<cluster-health-check-port from ConfigMap>"
targetPort: "<cluster-health-check-port from ConfigMap>"
name: public-health-check-port
protocol: TCP
type: LoadBalancer

11
k8s/nginx-https/container/Dockerfile Normal file
View File

@ -0,0 +1,11 @@
FROM nginx:1.13.1
LABEL maintainer "dev@bigchaindb.com"
WORKDIR /
RUN apt-get update \
&& apt-get -y upgrade \
&& apt-get autoremove \
&& apt-get clean
COPY nginx.conf.template /etc/nginx/nginx.conf
COPY nginx_entrypoint.bash /
EXPOSE 80 443 27017
ENTRYPOINT ["/nginx_entrypoint.bash"]

15
k8s/nginx-https/container/README.md Normal file
View File

@ -0,0 +1,15 @@
## Nginx container for Secure WebSocket Support
### Step 1: Build and Push the Latest Container
Use the `docker_build_and_push.bash` script to build the latest docker image
and upload it to Docker Hub.
Ensure that the image tag is updated to a new version number to properly
reflect any changes made to the container.
### Note about testing Websocket connections:
You can test the WebSocket server by using
[wsc](https://www.npmjs.com/package/wsc) tool with a command like:
`wsc -er wss://localhost:9985/api/v1/streams/valid_transactions`.

5
k8s/nginx-https/container/docker_build_and_push.bash Executable file
View File

@ -0,0 +1,5 @@
#!/bin/bash
docker build -t bigchaindb/nginx_https:1.0 .
docker push bigchaindb/nginx_https:1.0

193
k8s/nginx-https/container/nginx.conf.template Normal file
View File

@ -0,0 +1,193 @@
# Frontend API server that:
# 1. Acts as the HTTPS termination point.
# 2. Forwards BDB HTTP requests to OpenResty backend.
# 3. Forwards BDB WS requests to BDB backend.
# 4. Forwards MDB TCP connections to MDB backend.
# 5. Forwards requests from 3scale to OpenResty backend.
# 6. Does health check with LB.
worker_processes 2;
daemon off;
user nobody nogroup;
pid /tmp/nginx.pid;
error_log /dev/stderr;
events {
# Each worker handles up to 512 connections. Increase this for heavy
# workloads.
worker_connections 512;
accept_mutex on;
use epoll;
}
http {
access_log /dev/stdout combined buffer=16k flush=5s;
# Allow 10 req/sec from the same IP address, and store the counters in a
# `zone` or shared memory location tagged as 'one'.
limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;
# Enable logging when requests are being throttled.
limit_req_log_level notice;
# HTTP status code that is returned to the client; 429 is for TooManyRequests,
# ref. RFC 6585
limit_req_status 429;
# Limit requests from the same client, allow `burst` to 20 r/s,
# `nodelay` or drop connection immediately in case it exceeds this
# threshold.
limit_req zone=one burst=20 nodelay;
# `slowloris` attack mitigation settings.
client_body_timeout 10s;
client_header_timeout 10s;
# DNS resolver to use for all the backend names specified in this configuration.
resolver DNS_SERVER valid=30s ipv6=off;
keepalive_timeout 60s;
# The following map blocks enable lazy-binding to the backend at runtime,
# rather than binding as soon as NGINX starts.
map $remote_addr $bdb_backend {
default BIGCHAINDB_BACKEND_HOST;
}
map $remote_addr $openresty_backend {
default OPENRESTY_BACKEND_HOST;
}
# Frontend server for the external clients; acts as HTTPS termination point.
server {
listen CLUSTER_FRONTEND_PORT ssl;
server_name "CLUSTER_FQDN";
ssl_certificate /etc/nginx/ssl/cert.pem;
ssl_certificate_key /etc/nginx/ssl/cert.key;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5;
underscores_in_headers on;
# Forward websockets directly to backend BDB.
location /api/v1/streams/valid_transactions {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://$bdb_backend:BIGCHAINDB_WS_PORT;
proxy_read_timeout 600s;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
# Forward other URL paths as per business logic/use case to BDB or
# OpenResty instance.
location / {
proxy_ignore_client_abort on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# max client request body size: avg transaction size.
client_max_body_size 15k;
# No auth for GETs, forward directly to BDB.
if ($request_method = GET) {
proxy_pass http://$bdb_backend:BIGCHAINDB_API_PORT;
}
# POST requests get forwarded to OpenResty instance. Enable CORS too.
if ($request_method = POST ) {
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
add_header 'Access-Control-Expose-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
proxy_pass http://$openresty_backend:OPENRESTY_BACKEND_PORT;
}
# OPTIONS requests handling for CORS.
if ($request_method = 'OPTIONS') {
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range,app_key,app_id';
add_header 'Access-Control-Max-Age' 43200;
add_header 'Content-Type' 'text/plain charset=UTF-8';
add_header 'Content-Length' 0;
return 204;
}
}
}
# Frontend server for the load balancer to respond to health checks.
server {
listen HEALTH_CHECK_PORT;
location = /health {
return 200;
}
}
# Frontend server for the external clients; returns a pretty error message
# when an HTTP request is sent instead of HTTPS.
server {
listen 80;
server_name "CLUSTER_FQDN";
location / {
add_header Upgrade "TLS/1.2, HTTP/1.1" always;
default_type text/plain;
return 426 'Consider using the HTTPS protocol next time!';
}
}
# Frontend server to respond to requests from 3scale; forward directly to
# OpenResty instance.
server {
sendfile on;
listen THREESCALE_API_PORT;
location / {
proxy_pass http://$openresty_backend:OPENRESTY_BACKEND_PORT;
}
}
}
# NGINX stream block for TCP and UDP proxies. Used to proxy MDB TCP
# connection.
stream {
log_format mdb_log '[$time_iso8601] $realip_remote_addr $remote_addr '
'$proxy_protocol_addr $proxy_protocol_port '
'$protocol $status $session_time $bytes_sent '
'$bytes_received "$upstream_addr" "$upstream_bytes_sent" '
'"$upstream_bytes_received" "$upstream_connect_time" ';
access_log /dev/stdout mdb_log buffer=16k flush=5s;
# Define a zone 'two' of size 10 megabytes to store the counters
# that hold number of TCP connections from a specific IP address.
limit_conn_zone $binary_remote_addr zone=two:10m;
# Enable logging when connections are being throttled.
limit_conn_log_level notice;
# Allow 16 connections from the same IP address.
limit_conn two 16;
# DNS resolver to use for all the backend names specified in this configuration.
resolver DNS_SERVER valid=30s ipv6=off;
# The following map block enables lazy-binding to the backend at runtime,
# rather than binding as soon as NGINX starts.
map $remote_addr $mdb_backend {
default MONGODB_BACKEND_HOST;
}
# Frontend server to forward connections to MDB instance.
server {
listen MONGODB_FRONTEND_PORT so_keepalive=10m:1m:5;
preread_timeout 30s;
tcp_nodelay on;
proxy_pass $mdb_backend:MONGODB_BACKEND_PORT;
}
}

70
k8s/nginx-https/container/nginx_entrypoint.bash Executable file
View File

@ -0,0 +1,70 @@
#!/bin/bash
set -euo pipefail
# Cluster vars
cluster_fqdn=`printenv CLUSTER_FQDN`
cluster_frontend_port=`printenv CLUSTER_FRONTEND_PORT`
# NGINX vars
dns_server=`printenv DNS_SERVER`
health_check_port=`printenv HEALTH_CHECK_PORT`
# MongoDB vars
mongo_frontend_port=`printenv MONGODB_FRONTEND_PORT`
mongo_backend_host=`printenv MONGODB_BACKEND_HOST`
mongo_backend_port=`printenv MONGODB_BACKEND_PORT`
# OpenResty vars
openresty_backend_host=`printenv OPENRESTY_BACKEND_HOST`
openresty_backend_port=`printenv OPENRESTY_BACKEND_PORT`
threescale_api_port=`printenv THREESCALE_API_PORT`
# BigchainDB vars
bdb_backend_host=`printenv BIGCHAINDB_BACKEND_HOST`
bdb_api_port=`printenv BIGCHAINDB_API_PORT`
bdb_ws_port=`printenv BIGCHAINDB_WS_PORT`
# sanity check
if [[ -z "${cluster_frontend_port}" || \
-z "${mongo_frontend_port}" || \
-z "${mongo_backend_host}" || \
-z "${mongo_backend_port}" || \
-z "${openresty_backend_port}" || \
-z "${openresty_backend_host}" || \
-z "${threescale_api_port}" || \
-z "${bdb_backend_host}" || \
-z "${bdb_api_port}" || \
-z "${bdb_ws_port}" || \
-z "${dns_server}" || \
-z "${health_check_port}" || \
-z "${cluster_fqdn}" ]]; then
echo "Invalid environment settings detected. Exiting!"
exit 1
fi
NGINX_CONF_FILE=/etc/nginx/nginx.conf
# configure the nginx.conf file with env variables
sed -i "s|CLUSTER_FQDN|${cluster_fqdn}|g" ${NGINX_CONF_FILE}
sed -i "s|CLUSTER_FRONTEND_PORT|${cluster_frontend_port}|g" ${NGINX_CONF_FILE}
sed -i "s|MONGODB_FRONTEND_PORT|${mongo_frontend_port}|g" ${NGINX_CONF_FILE}
sed -i "s|MONGODB_BACKEND_HOST|${mongo_backend_host}|g" ${NGINX_CONF_FILE}
sed -i "s|MONGODB_BACKEND_PORT|${mongo_backend_port}|g" ${NGINX_CONF_FILE}
sed -i "s|OPENRESTY_BACKEND_PORT|${openresty_backend_port}|g" ${NGINX_CONF_FILE}
sed -i "s|OPENRESTY_BACKEND_HOST|${openresty_backend_host}|g" ${NGINX_CONF_FILE}
sed -i "s|THREESCALE_API_PORT|${threescale_api_port}|g" ${NGINX_CONF_FILE}
sed -i "s|BIGCHAINDB_BACKEND_HOST|${bdb_backend_host}|g" ${NGINX_CONF_FILE}
sed -i "s|BIGCHAINDB_API_PORT|${bdb_api_port}|g" ${NGINX_CONF_FILE}
sed -i "s|BIGCHAINDB_WS_PORT|${bdb_ws_port}|g" ${NGINX_CONF_FILE}
sed -i "s|DNS_SERVER|${dns_server}|g" ${NGINX_CONF_FILE}
sed -i "s|HEALTH_CHECK_PORT|${health_check_port}|g" ${NGINX_CONF_FILE}
# start nginx
echo "INFO: starting nginx..."
exec nginx -c /etc/nginx/nginx.conf

118
k8s/nginx-https/nginx-https-dep.yaml Normal file
View File

@ -0,0 +1,118 @@
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: ngx-https-instance-0-dep
spec:
replicas: 1
template:
metadata:
labels:
app: ngx-https-instance-0-dep
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx-https
image: bigchaindb/nginx_https:1.0
imagePullPolicy: Always
env:
- name: CLUSTER_FRONTEND_PORT
valueFrom:
configMapKeyRef:
name: vars
key: cluster-frontend-port
- name: HEALTH_CHECK_PORT
valueFrom:
configMapKeyRef:
name: vars
key: cluster-health-check-port
- name: CLUSTER_FQDN
valueFrom:
configMapKeyRef:
name: vars
key: cluster-fqdn
- name: DNS_SERVER
valueFrom:
configMapKeyRef:
name: vars
key: cluster-dns-server-ip
- name: MONGODB_FRONTEND_PORT
valueFrom:
configMapKeyRef:
name: vars
key: mongodb-frontend-port
- name: MONGODB_BACKEND_HOST
valueFrom:
configMapKeyRef:
name: vars
key: ngx-mdb-instance-name
- name: MONGODB_BACKEND_PORT
valueFrom:
configMapKeyRef:
name: vars
key: mongodb-backend-port
- name: OPENRESTY_BACKEND_PORT
valueFrom:
configMapKeyRef:
name: vars
key: openresty-backend-port
- name: OPENRESTY_BACKEND_HOST
valueFrom:
configMapKeyRef:
name: vars
key: openresty-backend-host
- name: THREESCALE_API_PORT
valueFrom:
configMapKeyRef:
name: vars
key: threescale-api-port
- name: BIGCHAINDB_BACKEND_HOST
valueFrom:
configMapKeyRef:
name: vars
key: ngx-bdb-instance-name
- name: BIGCHAINDB_API_PORT
valueFrom:
configMapKeyRef:
name: vars
key: bigchaindb-api-port
- name: BIGCHAINDB_WS_PORT
valueFrom:
configMapKeyRef:
name: vars
key: bigchaindb-ws-port
ports:
# return a pretty error message on port 80, since we are expecting
# HTTPS traffic.
- containerPort: 80
protocol: TCP
- containerPort: "<mongodb-frontend-port from ConfigMap>"
protocol: TCP
- containerPort: "<cluster-frontend-port from ConfigMap>"
protocol: TCP
- containerPort: "<threescale-api-port from ConfigMap>"
protocol: TCP
- containerPort: "<cluster-health-check-port from ConfigMap>"
protocol: TCP
name: ngx-port
livenessProbe:
httpGet:
path: /health
port: ngx-port
initialDelaySeconds: 15
periodSeconds: 15
failureThreshold: 3
timeoutSeconds: 10
resources:
limits:
cpu: 200m
memory: 768Mi
volumeMounts:
- name: https-certs
mountPath: /etc/nginx/ssl/
readOnly: true
restartPolicy: Always
volumes:
- name: https-certs
secret:
secretName: https-certs
defaultMode: 0400

32
k8s/nginx-https/nginx-https-svc.yaml Normal file
View File

@ -0,0 +1,32 @@
apiVersion: v1
kind: Service
metadata:
name: ngx-https-instance-0
namespace: default
labels:
name: ngx-https-instance-0
annotations:
# NOTE: the following annotation is a beta feature and
# only available in GCE/GKE and Azure as of now
service.beta.kubernetes.io/external-traffic: OnlyLocal
spec:
selector:
app: ngx-https-instance-0-dep
ports:
- port: "<cluster-frontend-port from ConfigMap>"
targetPort: "<cluster-frontend-port from ConfigMap>"
name: public-secure-cluster-port
protocol: TCP
- port: "<mongodb-frontend-port from ConfigMap>"
targetPort: "<mongodb-frontend-port from ConfigMap>"
name: public-mdb-port
protocol: TCP
- port: "<threescale-api-port from ConfigMap>"
targetPort: "<threescale-api-port from ConfigMap>"
name: public-threescale-port
protocol: TCP
- port: 80
targetPort: 80
name: public-insecure-cluster-port
protocol: TCP
type: LoadBalancer

64
k8s/nginx-openresty/nginx-openresty-dep.yaml Normal file
View File

@ -0,0 +1,64 @@
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: openresty-instance-0-dep
spec:
replicas: 1
template:
metadata:
labels:
app: openresty-instance-0-dep
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx-openresty
image: bigchaindb/nginx_3scale:2.0
imagePullPolicy: Always
env:
- name: DNS_SERVER
valueFrom:
configMapKeyRef:
name: vars
key: cluster-dns-server-ip
- name: OPENRESTY_FRONTEND_PORT
valueFrom:
configMapKeyRef:
name: vars
key: openresty-backend-port
- name: BIGCHAINDB_BACKEND_HOST
valueFrom:
configMapKeyRef:
name: vars
key: ngx-bdb-instance-name
- name: BIGCHAINDB_API_PORT
valueFrom:
configMapKeyRef:
name: vars
key: bigchaindb-api-port
ports:
- containerPort: "<openresty-backend-port from ConfigMap>"
protocol: TCP
name: openresty-port
volumeMounts:
- name: threescale-credentials
mountPath: /usr/local/openresty/nginx/conf/threescale
readOnly: true
livenessProbe:
httpGet:
path: /
port: openresty-port
initialDelaySeconds: 15
periodSeconds: 15
successThreshold: 1
failureThreshold: 3
timeoutSeconds: 10
resources:
limits:
cpu: 200m
memory: 768Mi
restartPolicy: Always
volumes:
- name: threescale-credentials
secret:
secretName: threescale-credentials
defaultMode: 0400

22
k8s/nginx-openresty/nginx-openresty-svc.yaml Normal file
View File

@ -0,0 +1,22 @@
apiVersion: v1
kind: Service
metadata:
name: openresty-instance-0
namespace: default
labels:
name: openresty-instance-0
annotations:
# NOTE: the following annotation is a beta feature and
# only available in GCE/GKE and Azure as of now
# Ref: https://kubernetes.io/docs/tutorials/services/source-ip/
service.beta.kubernetes.io/external-traffic: OnlyLocal
spec:
selector:
app: openresty-instance-0-dep
ports:
- port: "<openresty-backend-port from ConfigMap>"
targetPort: "<openresty-backend-port from ConfigMap>"
name: openresty-svc-port
protocol: TCP
type: ClusterIP
clusterIP: None

78
k8s/nginx/container/README.md
View File

@ -1,78 +0,0 @@
## Custom Nginx container for a Node
### Need
* Since, BigchainDB and MongoDB both need to expose ports to the outside
world (inter and intra cluster), we need to have a basic DDoS mitigation
strategy to ensure that we can provide proper uptime and security these
core services.
* We can have a proxy like nginx/haproxy in every node that listens to
global connections and applies cluster level entry policy.
### Implementation
* For MongoDB cluster communication, we will use nginx with an environment
variable specifying a ":" separated list of IPs in the whitelist. This list
contains the IPs of exising instances in the MongoDB replica set so as to
allow connections from the whitelist and avoid a DDoS.
* For BigchainDB connections, nginx needs to have rules to throttle
connections that are using resources over a threshold.
### Step 1: Build the Latest Container
Run `docker build -t bigchaindb/nginx:1.0 .` from this folder.
Optional: Upload container to Docker Hub:
`docker push bigchaindb/nginx:1.0`
### Step 2: Run the Container
Note that the whilelist IPs must be specified with the subnet in the CIDR
format, eg: `1.2.3.4/16`
```
docker run \
--env "MONGODB_FRONTEND_PORT=<port where nginx listens for MongoDB connections>" \
--env "MONGODB_BACKEND_HOST=<ip/hostname of instance where MongoDB is running>" \
--env "MONGODB_BACKEND_PORT=<port where MongoDB is listening for connections>" \
--env "BIGCHAINDB_FRONTEND_PORT=<port where nginx listens for BigchainDB connections>" \
--env "BIGCHAINDB_BACKEND_HOST=<ip/hostname of instance where BigchainDB is running>" \
--env "BIGCHAINDB_BACKEND_PORT=<port where BigchainDB is listening for connections>" \
--env "BIGCHAINDB_WS_BACKEND_PORT=<port where BigchainDB is listening for websocket connections>" \
--env "BIGCHAINDB_WS_FRONTEND_PORT=<port where nginx listens for BigchainDB WebSocket connections>" \
--env "MONGODB_WHITELIST=<a ':' separated list of IPs that can connect to MongoDB>" \
--env "DNS_SERVER=<ip of the dns server>" \
--name=ngx \
--publish=<port where nginx listens for MongoDB connections as specified above>:<correcponding host port> \
--publish=<port where nginx listens for BigchainDB connections as specified above>:<corresponding host port> \
--rm=true \
bigchaindb/nginx:1.0
```
For example:
```
docker run \
--env="MONGODB_FRONTEND_PORT=17017" \
--env="MONGODB_BACKEND_HOST=localhost" \
--env="MONGODB_BACKEND_PORT=27017" \
--env="BIGCHAINDB_FRONTEND_PORT=80" \
--env="BIGCHAINDB_BACKEND_HOST=localhost" \
--env="BIGCHAINDB_BACKEND_PORT=9984" \
--env="BIGCHAINDB_WS_FRONTEND_PORT=81" \
--env="BIGCHAINDB_WS_BACKEND_PORT=9985" \
--env="MONGODB_WHITELIST=192.168.0.0/16:10.0.2.0/24" \
--env="DNS_SERVER=127.0.0.1" \
--name=ngx \
--publish=80:80 \
--publish=17017:17017 \
--rm=true \
bigchaindb/nginx:1.0
```
### Note:
You can test the WebSocket server by using
[wsc](https://slack-redir.net/link?url=https%3A%2F%2Fwww.npmjs.com%2Fpackage%2Fwsc) tool with a command like:
`wsc -er ws://localhost:9985/api/v1/streams/valid_tx`.

127
k8s/nginx/container/nginx.conf.template
View File

@ -1,127 +0,0 @@
worker_processes 2;
daemon off;
user nobody nogroup;
pid /tmp/nginx.pid;
error_log /etc/nginx/nginx.error.log;
events {
worker_connections 256;
accept_mutex on;
use epoll;
}
http {
server_names_hash_bucket_size 128;
access_log /etc/nginx/nginx.access.log combined buffer=16k flush=5s;
# allow 10 req/sec from the same IP address, and store the counters in a
# `zone` or shared memory location tagged as 'one'.
limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;
# enable logging when requests are being throttled
limit_req_log_level notice;
# the http status code to return to the client when throttling;
# 429 is for TooManyRequests,
# ref. RFC 6585
limit_req_status 429;
resolver DNS_SERVER valid=20s;
map $remote_addr $bdb_backend {
default BIGCHAINDB_BACKEND_HOST;
}
server {
listen BIGCHAINDB_FRONTEND_PORT;
# server_name "FRONTEND_DNS_NAME";
underscores_in_headers on;
# max client request body size: avg transaction size
client_max_body_size 15k;
# keepalive connection settings
keepalive_timeout 20s;
# `slowloris` attack mitigation settings
client_body_timeout 10s;
client_header_timeout 10s;
location / {
proxy_ignore_client_abort on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_redirect off;
# TODO proxy_set_header X-Forwarded-Proto https;
# limit requests from the same client, allow `burst` to 20 r/s,
# `nodelay` or drop connection immediately in case it exceeds this
# threshold.
limit_req zone=one burst=20 nodelay;
proxy_pass http://$bdb_backend:BIGCHAINDB_BACKEND_PORT;
}
}
server {
listen BIGCHAINDB_WS_FRONTEND_PORT;
underscores_in_headers on;
# keepalive connection settings
keepalive_timeout 20s;
# `slowloris` attack mitigation settings
client_body_timeout 10s;
client_header_timeout 10s;
location /api/v1/streams/ {
proxy_read_timeout 300s;
proxy_pass http://$bdb_backend:BIGCHAINDB_WS_BACKEND_PORT;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
}
# NGINX stream block for TCP and UDP proxies
stream {
log_format mdb_log '[$time_iso8601] $realip_remote_addr $remote_addr '
'$proxy_protocol_addr $proxy_protocol_port '
'$protocol $status $session_time $bytes_sent '
'$bytes_received "$upstream_addr" "$upstream_bytes_sent" '
'"$upstream_bytes_received" "$upstream_connect_time" ';
access_log /etc/nginx/nginx.stream.access.log mdb_log buffer=16k flush=5s;
# define a zone 'two' of size 10 megabytes to store the counters
# that hold number of TCP connections from a specific IP address
limit_conn_zone $binary_remote_addr zone=two:10m;
# enable logging when connections are being throttled
limit_conn_log_level notice;
resolver DNS_SERVER valid=20s;
map $remote_addr $mdb_backend {
default MONGODB_BACKEND_HOST;
}
server {
listen MONGODB_FRONTEND_PORT so_keepalive=10m:1m:5;
preread_timeout 30s;
tcp_nodelay on;
# whitelist
#MONGODB_WHITELIST
allow all;
# deny access to everyone else
deny all;
# allow 16 connections from the same IP address
limit_conn two 16;
proxy_pass $mdb_backend:MONGODB_BACKEND_PORT;
}
}

53
k8s/nginx/container/nginx_entrypoint.bash
View File

@ -1,53 +0,0 @@
#!/bin/bash
set -euo pipefail
mongo_frontend_port=`printenv MONGODB_FRONTEND_PORT`
mongo_backend_host=`printenv MONGODB_BACKEND_HOST`
mongo_backend_port=`printenv MONGODB_BACKEND_PORT`
bdb_frontend_port=`printenv BIGCHAINDB_FRONTEND_PORT`
bdb_backend_host=`printenv BIGCHAINDB_BACKEND_HOST`
bdb_backend_port=`printenv BIGCHAINDB_BACKEND_PORT`
bdb_ws_frontend_port=`printenv BIGCHAINDB_WS_FRONTEND_PORT`
bdb_ws_backend_port=`printenv BIGCHAINDB_WS_BACKEND_PORT`
mongo_whitelist=`printenv MONGODB_WHITELIST`
dns_server=`printenv DNS_SERVER`
# sanity checks
if [[ -z "${mongo_frontend_port}" || \
-z "${mongo_backend_host}" || \
-z "${mongo_backend_port}" || \
-z "${bdb_frontend_port}" || \
-z "${bdb_backend_host}" || \
-z "${bdb_backend_port}" || \
-z "${bdb_ws_backend_port}" || \
-z "${bdb_ws_frontend_port}" || \
-z "${dns_server}" ]] ; then
echo "Invalid environment settings detected. Exiting!"
exit 1
fi
NGINX_CONF_FILE=/etc/nginx/nginx.conf
# configure the nginx.conf file with env variables
sed -i "s|MONGODB_FRONTEND_PORT|${mongo_frontend_port}|g" $NGINX_CONF_FILE
sed -i "s|MONGODB_BACKEND_HOST|${mongo_backend_host}|g" $NGINX_CONF_FILE
sed -i "s|MONGODB_BACKEND_PORT|${mongo_backend_port}|g" $NGINX_CONF_FILE
sed -i "s|BIGCHAINDB_FRONTEND_PORT|${bdb_frontend_port}|g" $NGINX_CONF_FILE
sed -i "s|BIGCHAINDB_BACKEND_HOST|${bdb_backend_host}|g" $NGINX_CONF_FILE
sed -i "s|BIGCHAINDB_BACKEND_PORT|${bdb_backend_port}|g" $NGINX_CONF_FILE
sed -i "s|BIGCHAINDB_WS_FRONTEND_PORT|${bdb_ws_frontend_port}|g" $NGINX_CONF_FILE
sed -i "s|BIGCHAINDB_WS_BACKEND_PORT|${bdb_ws_backend_port}|g" $NGINX_CONF_FILE
sed -i "s|DNS_SERVER|${dns_server}|g" $NGINX_CONF_FILE
# populate the whitelist in the conf file as per MONGODB_WHITELIST env var
hosts=$(echo ${mongo_whitelist} | tr ":" "\n")
for host in $hosts; do
sed -i "s|MONGODB_WHITELIST|allow ${host};\n MONGODB_WHITELIST|g" $NGINX_CONF_FILE
done
# remove the MONGODB_WHITELIST marker string from template
sed -i "s|MONGODB_WHITELIST||g" $NGINX_CONF_FILE
# start nginx
echo "INFO: starting nginx..."
exec nginx -c /etc/nginx/nginx.conf

67
k8s/nginx/nginx-dep.yaml
View File

@ -1,67 +0,0 @@
###############################################################
# This config file runs nginx as a k8s deployment and exposes #
# it using an external load balancer. #
# This deployment is used as a front end to both BigchainDB #
# and MongoDB. #
###############################################################
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: ngx-instance-0-dep
spec:
replicas: 1
template:
metadata:
labels:
app: ngx-instance-0-dep
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx
image: bigchaindb/nginx:1.0
imagePullPolicy: Always
env:
- name: MONGODB_FRONTEND_PORT
value: "27017"
- name: MONGODB_BACKEND_HOST
value: mdb-instance-0.default.svc.cluster.local
- name: MONGODB_BACKEND_PORT
value: "27017"
- name: BIGCHAINDB_FRONTEND_PORT
value: "80"
- name: BIGCHAINDB_BACKEND_HOST
value: bdb-instance-0.default.svc.cluster.local
- name: BIGCHAINDB_BACKEND_PORT
value: "9984"
- name: BIGCHAINDB_WS_FRONTEND_PORT
value: "81"
- name: BIGCHAINDB_WS_BACKEND_PORT
value: "9985"
- name: DNS_SERVER
value: "10.0.0.10"
- name: MONGODB_WHITELIST
valueFrom:
configMapKeyRef:
name: mongodb-whitelist
key: allowed-hosts
ports:
- containerPort: 27017
hostPort: 27017
name: public-mdb-port
protocol: TCP
- containerPort: 80
hostPort: 80
name: public-bdb-port
protocol: TCP
- containerPort: 81
hostPort: 81
name: public-ws-port
protocol: TCP
resources:
limits:
cpu: 200m
memory: 768Mi
#livenessProbe: TODO(Krish)
#readinessProbe: TODO(Krish)
restartPolicy: Always

28
k8s/nginx/nginx-svc.yaml
View File

@ -1,28 +0,0 @@
apiVersion: v1
kind: Service
metadata:
name: ngx-instance-0
namespace: default
labels:
name: ngx-instance-0
annotations:
# NOTE: the following annotation is a beta feature and
# only available in GCE/GKE and Azure as of now
service.beta.kubernetes.io/external-traffic: OnlyLocal
spec:
selector:
app: ngx-instance-0-dep
ports:
- port: 27017
targetPort: 27017
name: ngx-public-mdb-port
protocol: TCP
- port: 80
targetPort: 80
name: ngx-public-api-port
protocol: TCP
- port: 81
targetPort: 81
name: ngx-public-ws-port
protocol: TCP
type: LoadBalancer