diff --git a/docs/server/source/production-deployment-template/add-node-on-kubernetes.rst b/docs/server/source/production-deployment-template/add-node-on-kubernetes.rst index fd0611e6..b4444d89 100644 --- a/docs/server/source/production-deployment-template/add-node-on-kubernetes.rst +++ b/docs/server/source/production-deployment-template/add-node-on-kubernetes.rst @@ -71,10 +71,10 @@ Step 2: Prepare the New Kubernetes Cluster Follow the steps in the sections to set up Storage Classes and Persistent Volume Claims, and to run MongoDB in the new cluster: -1. :ref:`Add Storage Classes `. -2. :ref:`Add Persistent Volume Claims `. +1. :ref:`Add Storage Classes `. +2. :ref:`Add Persistent Volume Claims `. 3. :ref:`Create the Config Map `. -4. :ref:`Run MongoDB instance `. +4. :ref:`Run MongoDB instance `. Step 3: Add the New MongoDB Instance to the Existing Replica Set @@ -166,13 +166,13 @@ show-config`` command to check that the keyring is updated. Step 7: Run NGINX as a Deployment --------------------------------- -Please see :ref:`this page ` to +Please see :ref:`this page ` to set up NGINX in your new node. Step 8: Test Your New BigchainDB Node ------------------------------------- -Please refer to the testing steps :ref:`here ` to verify that your new BigchainDB node is working as expected. diff --git a/docs/server/source/production-deployment-template/index.rst b/docs/server/source/production-deployment-template/index.rst index 766c69b9..d47c463a 100644 --- a/docs/server/source/production-deployment-template/index.rst +++ b/docs/server/source/production-deployment-template/index.rst @@ -20,9 +20,9 @@ Feel free change things to suit your needs or preferences. revoke-tls-certificate template-kubernetes-azure node-on-kubernetes - add-node-on-kubernetes - upgrade-on-kubernetes - log-analytics - easy-rsa - cloud-manager node-config-map-and-secrets + log-analytics + cloud-manager + easy-rsa + upgrade-on-kubernetes + add-node-on-kubernetes diff --git a/docs/server/source/production-deployment-template/node-config-map-and-secrets.rst b/docs/server/source/production-deployment-template/node-config-map-and-secrets.rst index 8a62f185..d20a65dc 100644 --- a/docs/server/source/production-deployment-template/node-config-map-and-secrets.rst +++ b/docs/server/source/production-deployment-template/node-config-map-and-secrets.rst @@ -29,6 +29,39 @@ where all data values must be base64-encoded. This is true of all Kubernetes ConfigMaps and Secrets.) + +vars.cluster-fqdn +~~~~~~~~~~~~~~~~~ + +The ``cluster-fqdn`` field specifies the domain you would have +:ref:`registered before <2. Register a Domain and Get an SSL Certificate for It>`. + + +vars.cluster-frontend-port +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``cluster-frontend-port`` field specifies the port on which your cluster +will be available to all external clients. +It is set to the HTTPS port ``443`` by default. + + +vars.cluster-health-check-port +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``cluster-healthcheck-port`` is the port number on which health check +probes are sent to the main NGINX instance. +It is set to ``8888`` by default. + + +vars.cluster-dns-server-ip +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``cluster-dns-server-ip`` is the IP of the DNS server for a node. +We use DNS for service discovery. A Kubernetes deployment always has a DNS +server (``kube-dns``) running at 10.0.0.10, and since we use Kubernetes, this is +set to ``10.0.0.10`` by default, which is the default ``kube-dns`` IP address. + + vars.mdb-instance-name and Similar ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -49,6 +82,57 @@ There are some things worth noting about the ``mdb-instance-name``: * We use ``mdb-instance-0``, ``mdb-instance-1`` and so on in our documentation. Your BigchainDB cluster may use a different naming convention. + +vars.ngx-ndb-instance-name and Similar +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +NGINX needs the FQDN of the servers inside the cluster to be able to forward +traffic. +The ``ngx-openresty-instance-name``, ``ngx-mdb-instance-name`` and +``ngx-bdb-instance-name`` are the FQDNs of the OpenResty instance, the MongoDB +instance, and the BigchainDB instance in this Kubernetes cluster respectively. +In Kubernetes, this is usually the name of the module specified in the +corresponding ``vars.*-instance-name`` followed by the +``.svc.cluster.local``. For example, if you run OpenResty in +the default Kubernetes namespace, this will be +``.default.svc.cluster.local`` + + +vars.mongodb-frontend-port and vars.mongodb-backend-port +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``mongodb-frontend-port`` is the port number on which external clients can +access MongoDB. This needs to be restricted to only other MongoDB instances +by enabling an authentication mechanism on MongoDB cluster. +It is set to ``27017`` by default. + +The ``mongodb-backend-port`` is the port number on which MongoDB is actually +available/listening for requests in your cluster. +It is also set to ``27017`` by default. + + +vars.openresty-backend-port +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``openresty-backend-port`` is the port number on which OpenResty is +listening for requests. +This is used by the NGINX instance to forward requests +destined for the OpenResty instance to the right port. +This is also used by OpenResty instance to bind to the correct port to +receive requests from NGINX instance. +It is set to ``80`` by default. + + +vars.bigchaindb-api-port and vars.bigchaindb-ws-port +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``bigchaindb-api-port`` is the port number on which BigchainDB is +listening for HTTP requests. Currently set to ``9984`` by default. + +The ``bigchaindb-ws-port`` is the port number on which BigchainDB is +listening for Websocket requests. Currently set to ``9985`` by default. + + bdb-config.bdb-keyring ~~~~~~~~~~~~~~~~~~~~~~~ @@ -127,12 +211,10 @@ If you're not using 3scale, you can delete the ``threescale-credentials`` Secret or leave all the values blank (``""``). -If you *are* using 3scale, you can get the value for ``frontend-api-dns-name`` -using something like ``echo "your.nodesubdomain.net" | base64 -w 0`` - -To get the values for ``secret-token``, ``service-id``, -``version-header`` and ``provider-key``, login to your 3scale admin, -then click **APIs** and click on **Integration** for the relevant API. +If you *are* using 3scale, get the values for ``secret-token``, +``service-id``, ``version-header`` and ``provider-key`` by logging in to 3scale +portal using your admin account, click **APIs** and click on **Integration** +for the relevant API. Scroll to the bottom of the page and click the small link in the lower right corner, labelled **Download the NGINX Config files**. You'll get a ``.zip`` file. diff --git a/docs/server/source/production-deployment-template/node-on-kubernetes.rst b/docs/server/source/production-deployment-template/node-on-kubernetes.rst index 3ee7d2c5..7f0ffc01 100644 --- a/docs/server/source/production-deployment-template/node-on-kubernetes.rst +++ b/docs/server/source/production-deployment-template/node-on-kubernetes.rst @@ -113,46 +113,40 @@ Step 4: Start the NGINX Service public IP to be assigned. * You have the option to use vanilla NGINX without HTTPS support or an - OpenResty NGINX integrated with 3scale API Gateway. + NGINX with HTTPS support integrated with 3scale API Gateway. Step 4.1: Vanilla NGINX ^^^^^^^^^^^^^^^^^^^^^^^ - * This configuration is located in the file ``nginx/nginx-svc.yaml``. + * This configuration is located in the file ``nginx-http/nginx-http-svc.yaml``. * Set the ``metadata.name`` and ``metadata.labels.name`` to the value set in ``ngx-instance-name`` in the ConfigMap above. * Set the ``spec.selector.app`` to the value set in ``ngx-instance-name`` in the ConfigMap followed by ``-dep``. For example, if the value set in the - ``ngx-instance-name`` is ``ngx-instance-0``, set the - ``spec.selector.app`` to ``ngx-instance-0-dep``. + ``ngx-instance-name`` is ``ngx-http-instance-0``, set the + ``spec.selector.app`` to ``ngx-http-instance-0-dep``. - * Set ``ngx-public-mdb-port.port`` to 27017, or the port number on which you - want to expose MongoDB service. - Set the ``ngx-public-mdb-port.targetPort`` to the port number on which the - Kubernetes MongoDB service will be present. + * Set ``ports[0].port`` and ``ports[0].targetPort`` to the value set in the + ``cluster-frontend-port`` in the ConfigMap above. This is the + ``public-cluster-port`` in the file which is the ingress in to the cluster. - * Set ``ngx-public-api-port.port`` to 80, or the port number on which you want to - expose BigchainDB API service. - Set the ``ngx-public-api-port.targetPort`` to the port number on which the - Kubernetes BigchainDB API service will present. + * Set ``ports[1].port`` and ``ports[1].targetPort`` to the value set in the + ``cluster-health-check-port`` in the ConfigMap above. This is the + ``public-health-check-port`` in the file which is the health check port. + Note: This will be removed in the future. - * Set ``ngx-public-ws-port.port`` to 81, or the port number on which you want to - expose BigchainDB Websocket service. - Set the ``ngx-public-ws-port.targetPort`` to the port number on which the - BigchainDB Websocket service will be present. - * Start the Kubernetes Service: .. code:: bash - $ kubectl --context k8s-bdb-test-cluster-0 apply -f nginx/nginx-svc.yaml + $ kubectl --context k8s-bdb-test-cluster-0 apply -f nginx-http/nginx-http-svc.yaml -Step 4.2: OpenResty NGINX + 3scale -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Step 4.2: NGINX with HTTPS + 3scale +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * You have to enable HTTPS for this one and will need an HTTPS certificate for your domain. @@ -160,42 +154,42 @@ Step 4.2: OpenResty NGINX + 3scale * You should have already created the necessary Kubernetes Secrets in the previous step (e.g. ``https-certs`` and ``threescale-credentials``). - * This configuration is located in the file ``nginx-3scale/nginx-3scale-svc.yaml``. + * This configuration is located in the file ``nginx-https/nginx-https-svc.yaml``. * Set the ``metadata.name`` and ``metadata.labels.name`` to the value set in ``ngx-instance-name`` in the ConfigMap above. * Set the ``spec.selector.app`` to the value set in ``ngx-instance-name`` in the ConfigMap followed by ``-dep``. For example, if the value set in the - ``ngx-instance-name`` is ``ngx-instance-0``, set the - ``spec.selector.app`` to ``ngx-instance-0-dep``. + ``ngx-instance-name`` is ``ngx-https-instance-0``, set the + ``spec.selector.app`` to ``ngx-https-instance-0-dep``. - * Set ``ngx-public-mdb-port.port`` to 27017, or the port number on which you + * Set ``ports[0].port`` and ``ports[0].targetPort`` to the value set in the + ``cluster-frontend-port`` in the ConfigMap above. This is the + ``public-secure-cluster-port`` in the file which is the ingress in to the cluster. + + * Set ``ports[1].port`` and ``ports[1].targetPort`` to the value set in the + ``mongodb-frontend-port`` in the ConfigMap above. This is the + ``public-mdb-port`` in the file which specifies where MongoDB is + available. + + * Set ``ports[2].port`` and ``ports[2].targetPort`` to the value set in the + ``threescale-api-port`` in the ConfigMap above. This is the + ``public-threescale-port`` in the file which specifies where OpenResty is + available. + + * Set ``threescale-api-port`` to 27017, or the port number on which you want to expose MongoDB service. Set the ``ngx-public-mdb-port.targetPort`` to the port number on which the Kubernetes MongoDB service will be present. + Note: This is only used for testing with 3scale, and will be removed in + the future. - * Set ``ngx-public-3scale-port.port`` to 8080, or the port number on which - you want to let 3scale communicate with Openresty NGINX for authenctication. - Set the ``ngx-public-3scale-port.targetPort`` to the port number on which - this Openresty NGINX service will be listening to for communication with - 3scale. - - * Set ``ngx-public-bdb-port.port`` to 443, or the port number on which you want - to expose BigchainDB API service. - Set the ``ngx-public-api-port.targetPort`` to the port number on which the - Kubernetes BigchainDB API service will present. - - * Set ``ngx-public-bdb-port-http.port`` to 80, or the port number on which you - want to expose BigchainDB Websocket service. - Set the ``ngx-public-bdb-port-http.targetPort`` to the port number on which the - BigchainDB Websocket service will be present. - * Start the Kubernetes Service: .. code:: bash - $ kubectl --context k8s-bdb-test-cluster-0 apply -f nginx-3scale/nginx-3scale-svc.yaml + $ kubectl --context k8s-bdb-test-cluster-0 apply -f nginx-https/nginx-https-svc.yaml Step 5: Assign DNS Name to the NGINX Public IP @@ -246,7 +240,7 @@ Step 6: Start the MongoDB Kubernetes Service * Set the ``metadata.name`` and ``metadata.labels.name`` to the value set in ``mdb-instance-name`` in the ConfigMap above. - + * Set the ``spec.selector.app`` to the value set in ``mdb-instance-name`` in the ConfigMap followed by ``-ss``. For example, if the value set in the ``mdb-instance-name`` is ``mdb-instance-0``, set the @@ -266,7 +260,7 @@ Step 7: Start the BigchainDB Kubernetes Service * Set the ``metadata.name`` and ``metadata.labels.name`` to the value set in ``bdb-instance-name`` in the ConfigMap above. - + * Set the ``spec.selector.app`` to the value set in ``bdb-instance-name`` in the ConfigMap followed by ``-dep``. For example, if the value set in the ``bdb-instance-name`` is ``bdb-instance-0``, set the @@ -279,72 +273,77 @@ Step 7: Start the BigchainDB Kubernetes Service $ kubectl --context k8s-bdb-test-cluster-0 apply -f bigchaindb/bigchaindb-svc.yaml -Step 8: Start the NGINX Kubernetes Deployment +Step 8: Start the OpenResty Kubernetes Service +---------------------------------------------- + + * This configuration is located in the file ``nginx-openresty/nginx-openresty-svc.yaml``. + + * Set the ``metadata.name`` and ``metadata.labels.name`` to the value + set in ``openresty-instance-name`` in the ConfigMap above. + + * Set the ``spec.selector.app`` to the value set in ``openresty-instance-name`` in + the ConfigMap followed by ``-dep``. For example, if the value set in the + ``openresty-instance-name`` is ``openresty-instance-0``, set the + ``spec.selector.app`` to ``openresty-instance-0-dep``. + + * Start the Kubernetes Service: + + .. code:: bash + + $ kubectl --context k8s-bdb-test-cluster-0 apply -f nginx-openresty/nginx-openresty-svc.yaml + + +Step 9: Start the NGINX Kubernetes Deployment --------------------------------------------- - * NGINX is used as a proxy to both the BigchainDB and MongoDB instances in - the node. It proxies HTTP requests on port 80 to the BigchainDB backend, - and TCP connections on port 27017 to the MongoDB backend. + * NGINX is used as a proxy to OpenResty, BigchainDB and MongoDB instances in + the node. It proxies HTTP/HTTPS requests on the ``clusted-frontend-port`` + to the corresponding OpenResty or BigchainDB backend, and TCP connections + on ``mongodb-frontend-port`` to the MongoDB backend. - * As in step 4, you have the option to use vanilla NGINX or an OpenResty - NGINX integrated with 3scale API Gateway. + * As in step 4, you have the option to use vanilla NGINX without HTTPS or + NGINX with HTTPS support integrated with 3scale API Gateway. -Step 8.1: Vanilla NGINX +Step 9.1: Vanilla NGINX ^^^^^^^^^^^^^^^^^^^^^^^ - * This configuration is located in the file ``nginx/nginx-dep.yaml``. + * This configuration is located in the file ``nginx-http/nginx-http-dep.yaml``. * Set the ``metadata.name`` and ``spec.template.metadata.labels.app`` to the value set in ``ngx-instance-name`` in the ConfigMap followed by a ``-dep``. For example, if the value set in the ``ngx-instance-name`` is - ``ngx-instance-0``, set the fields to ``ngx-instance-0-dep``. + ``ngx-http-instance-0``, set the fields to ``ngx-http-instance-0-dep``. + + * Set the ports to be exposed from the pod in the + ``spec.containers[0].ports`` section. We currently expose 3 ports - + ``mongodb-frontend-port``, ``cluster-frontend-port`` and + ``cluster-health-check-port``. Set them to the values specified in the + ConfigMap. - * Set ``MONGODB_BACKEND_HOST`` env var to - the value set in ``mdb-instance-name`` in the ConfigMap, followed by - ``.default.svc.cluster.local``. For example, if the value set in the - ``mdb-instance-name`` is ``mdb-instance-0``, set the - ``MONGODB_BACKEND_HOST`` env var to - ``mdb-instance-0.default.svc.cluster.local``. - - * Set ``BIGCHAINDB_BACKEND_HOST`` env var to - the value set in ``bdb-instance-name`` in the ConfigMap, followed by - ``.default.svc.cluster.local``. For example, if the value set in the - ``bdb-instance-name`` is ``bdb-instance-0``, set the - ``BIGCHAINDB_BACKEND_HOST`` env var to - ``bdb-instance-0.default.svc.cluster.local``. - * Start the Kubernetes Deployment: .. code:: bash - $ kubectl --context k8s-bdb-test-cluster-0 apply -f nginx/nginx-dep.yaml + $ kubectl --context k8s-bdb-test-cluster-0 apply -f nginx-http/nginx-http-dep.yaml -Step 8.2: OpenResty NGINX + 3scale -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Step 9.2: NGINX with HTTPS + 3scale +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * This configuration is located in the file - ``nginx-3scale/nginx-3scale-dep.yaml``. + ``nginx-https/nginx-https-dep.yaml``. * Set the ``metadata.name`` and ``spec.template.metadata.labels.app`` to the value set in ``ngx-instance-name`` in the ConfigMap followed by a ``-dep``. For example, if the value set in the ``ngx-instance-name`` is - ``ngx-instance-0``, set the fields to ``ngx-instance-0-dep``. + ``ngx-https-instance-0``, set the fields to ``ngx-https-instance-0-dep``. + + * Set the ports to be exposed from the pod in the + ``spec.containers[0].ports`` section. We currently expose 4 ports - + ``mongodb-frontend-port``, ``cluster-frontend-port``, + ``threescale-api-port`` and ``cluster-health-check-port``. Set them to + the values specified in the ConfigMap. - * Set ``MONGODB_BACKEND_HOST`` env var to - the value set in ``mdb-instance-name`` in the ConfigMap, followed by - ``.default.svc.cluster.local``. For example, if the value set in the - ``mdb-instance-name`` is ``mdb-instance-0``, set the - ``MONGODB_BACKEND_HOST`` env var to - ``mdb-instance-0.default.svc.cluster.local``. - - * Set ``BIGCHAINDB_BACKEND_HOST`` env var to - the value set in ``bdb-instance-name`` in the ConfigMap, followed by - ``.default.svc.cluster.local``. For example, if the value set in the - ``bdb-instance-name`` is ``bdb-instance-0``, set the - ``BIGCHAINDB_BACKEND_HOST`` env var to - ``bdb-instance-0.default.svc.cluster.local``. - * Start the Kubernetes Deployment: .. code:: bash @@ -352,8 +351,8 @@ Step 8.2: OpenResty NGINX + 3scale $ kubectl --context k8s-bdb-test-cluster-0 apply -f nginx-3scale/nginx-3scale-dep.yaml -Step 9: Create Kubernetes Storage Classes for MongoDB ------------------------------------------------------ +Step 10: Create Kubernetes Storage Classes for MongoDB +------------------------------------------------------ MongoDB needs somewhere to store its data persistently, outside the container where MongoDB is running. @@ -425,7 +424,7 @@ Kubernetes just looks for a storageAccount with the specified skuName and location. -Step 10: Create Kubernetes Persistent Volume Claims +Step 11: Create Kubernetes Persistent Volume Claims --------------------------------------------------- Next, you will create two PersistentVolumeClaim objects ``mongo-db-claim`` and @@ -457,7 +456,7 @@ Initially, the status of persistent volume claims might be "Pending" but it should become "Bound" fairly quickly. -Step 11: Start a Kubernetes StatefulSet for MongoDB +Step 12: Start a Kubernetes StatefulSet for MongoDB --------------------------------------------------- * This configuration is located in the file ``mongodb/mongo-ss.yaml``. @@ -512,7 +511,7 @@ Step 11: Start a Kubernetes StatefulSet for MongoDB $ kubectl --context k8s-bdb-test-cluster-0 get pods -w -Step 12: Configure Users and Access Control for MongoDB +Step 13: Configure Users and Access Control for MongoDB ------------------------------------------------------- * In this step, you will create a user on MongoDB with authorization @@ -640,7 +639,7 @@ Step 12: Configure Users and Access Control for MongoDB } ) -Step 13: Start a Kubernetes Deployment for MongoDB Monitoring Agent +Step 14: Start a Kubernetes Deployment for MongoDB Monitoring Agent ------------------------------------------------------------------- * This configuration is located in the file @@ -661,7 +660,7 @@ Step 13: Start a Kubernetes Deployment for MongoDB Monitoring Agent $ kubectl --context k8s-bdb-test-cluster-0 apply -f mongodb-monitoring-agent/mongo-mon-dep.yaml -Step 14: Start a Kubernetes Deployment for MongoDB Backup Agent +Step 15: Start a Kubernetes Deployment for MongoDB Backup Agent --------------------------------------------------------------- * This configuration is located in the file @@ -682,7 +681,7 @@ Step 14: Start a Kubernetes Deployment for MongoDB Backup Agent $ kubectl --context k8s-bdb-test-cluster-0 apply -f mongodb-backup-agent/mongo-backup-dep.yaml -Step 15: Start a Kubernetes Deployment for BigchainDB +Step 16: Start a Kubernetes Deployment for BigchainDB ----------------------------------------------------- * This configuration is located in the file @@ -717,7 +716,35 @@ Step 15: Start a Kubernetes Deployment for BigchainDB * You can check its status using the command ``kubectl get deployments -w`` -Step 16: Configure the MongoDB Cloud Manager +Step 17: Start a Kubernetes Deployment for OpenResty +---------------------------------------------------- + + * This configuration is located in the file + ``nginx-openresty/nginx-openresty-dep.yaml``. + + * Set ``metadata.name`` and ``spec.template.metadata.labels.app`` to the + value set in ``openresty-instance-name`` in the ConfigMap, followed by + ``-dep``. + For example, if the value set in the + ``openresty-instance-name`` is ``openresty-instance-0``, set the fields to + the value ``openresty-instance-0-dep``. + + * Set the port to be exposed from the pod in the + ``spec.containers[0].ports`` section. We currently expose the port at + which OpenResty is listening for requests, ``openresty-backend-port`` in + the above ConfigMap. + + * Create the OpenResty Deployment using: + + .. code:: bash + + $ kubectl --context k8s-bdb-test-cluster-0 apply -f nginx-openresty/nginx-openresty-dep.yaml + + + * You can check its status using the command ``kubectl get deployments -w`` + + +Step 18: Configure the MongoDB Cloud Manager -------------------------------------------- Refer to the @@ -726,10 +753,10 @@ for details on how to configure the MongoDB Cloud Manager to enable monitoring and backup. -Step 17: Verify the BigchainDB Node Setup +Step 19: Verify the BigchainDB Node Setup ----------------------------------------- -Step 17.1: Testing Internally +Step 19.1: Testing Internally ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To test the setup of your BigchainDB node, you could use a Docker container @@ -773,38 +800,69 @@ To test the BigchainDB instance: $ nslookup bdb-instance-0 $ dig +noall +answer _bdb-port._tcp.bdb-instance-0.default.svc.cluster.local SRV + + $ dig +noall +answer _bdb-ws-port._tcp.bdb-instance-0.default.svc.cluster.local SRV $ curl -X GET http://bdb-instance-0:9984 + + $ wsc -er ws://bdb-instance-0:9985/api/v1/streams/valid_transactions + -To test the NGINX instance: +To test the OpenResty instance: + +.. code:: bash + + $ nslookup openresty-instance-0 + + $ dig +noall +answer _openresty-svc-port._tcp.openresty-instance-0.default.svc.cluster.local SRV + +To verify if OpenResty instance forwards the requests properly, send a ``POST`` +transaction to OpenResty at post ``80`` and check the response from the backend +BigchainDB instance. + + +To test the vanilla NGINX instance: .. code:: bash - $ nslookup ngx-instance-0 + $ nslookup ngx-http-instance-0 - $ dig +noall +answer _ngx-public-mdb-port._tcp.ngx-instance-0.default.svc.cluster.local SRV + $ dig +noall +answer _public-cluster-port._tcp.ngx-http-instance-0.default.svc.cluster.local SRV - $ dig +noall +answer _ngx-public-bdb-port._tcp.ngx-instance-0.default.svc.cluster.local SRV + $ dig +noall +answer _public-health-check-port._tcp.ngx-http-instance-0.default.svc.cluster.local SRV - $ curl -X GET http://ngx-instance-0:27017 + $ wsc -er ws://ngx-http-instance-0/api/v1/streams/valid_transactions -The curl command should result get the response -``curl: (7) Failed to connect to ngx-instance-0 port 27017: Connection refused``. + $ curl -X GET http://ngx-http-instance-0:27017 -If you ran the vanilla NGINX instance, run: +The above curl command should result in the response +``It looks like you are trying to access MongoDB over HTTP on the native driver port.`` + + + +To test the NGINX instance with HTTPS and 3scale integration: .. code:: bash + + $ nslookup ngx-https-instance-0 - $ curl -X GET http://ngx-instance-0:80 - -If you ran the OpenResty NGINX + 3scale instance, run: + $ dig +noall +answer _public-secure-cluster-port._.tcp.ngx-https-instance-0.default.svc.cluster.local SRV -.. code:: bash + $ dig +noall +answer _public-mdb-port._.tcp.ngx-https-instance-0.default.svc.cluster.local SRV - $ curl -X GET https://ngx-instance-0 + $ dig +noall +answer _public-threescale-port._.tcp.ngx-https-instance-0.default.svc.cluster.local SRV + + $ dig +noall +answer _public-insecure-cluster-port._.tcp.ngx-https-instance-0.default.svc.cluster.local SRV + + $ wsc -er wss://ngx-https-instance-0/api/v1/streams/valid_transactions + + $ curl -X GET http://ngx-https-instance-0:27017 + +The above curl command should result in the response +``It looks like you are trying to access MongoDB over HTTP on the native driver port.`` -Step 17.2: Testing Externally +Step 19.2: Testing Externally ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Check the MongoDB monitoring and backup agent on the MongoDB Cloud Manager @@ -816,3 +874,4 @@ server version, among other things. Use the Python Driver to send some transactions to the BigchainDB node and verify that your node or cluster works as expected. + diff --git a/docs/server/source/production-deployment-template/workflow.rst b/docs/server/source/production-deployment-template/workflow.rst index c511b8f9..d831287e 100644 --- a/docs/server/source/production-deployment-template/workflow.rst +++ b/docs/server/source/production-deployment-template/workflow.rst @@ -48,7 +48,8 @@ Similarly, other instances must also have unique names in the cluster. #. Name of the MongoDB instance (``mdb-instance-*``) #. Name of the BigchainDB instance (``bdb-instance-*``) -#. Name of the NGINX instance (``ngx-instance-*``) +#. Name of the NGINX instance (``ngx-http-instance-*`` or ``ngx-https-instance-*``) +#. Name of the OpenResty instance (``openresty-instance-*``) #. Name of the MongoDB monitoring agent instance (``mdb-mon-instance-*``) #. Name of the MongoDB backup agent instance (``mdb-bak-instance-*``) diff --git a/k8s/configuration/config-map.yaml b/k8s/configuration/config-map.yaml index 518ce1ff..776ceedd 100644 --- a/k8s/configuration/config-map.yaml +++ b/k8s/configuration/config-map.yaml @@ -7,25 +7,81 @@ metadata: name: vars namespace: default data: - # MongoDB + # cluster-fqdn is the DNS name registered for your HTTPS certificate. + cluster-fqdn: "bdb.example.com" + + # cluster-frontend-port is the port number on which this node's services + # are available to external clients. + cluster-frontend-port: "443" + + # cluster-health-check-port is the port number on which an external load + # balancer can check the status/liveness of the external/public server. + cluster-health-check-port: "8888" + + # cluster-dns-server-ip is the IP of the DNS server. A Kubernetes deployment + # always has a DNS server (kube-dns) running at 10.0.0.10 + cluster-dns-server-ip: "10.0.0.10" + + # mdb-instance-name is the name of the MongoDB instance in this cluster. mdb-instance-name: "" - # BigchainDB + + # ngx-instance-name is the name of the NGINX instance in this cluster. + ngx-instance-name: "" + + # openresty-instance-name is the name of the OpenResty instance in this + # cluster. + openresty-instance-name: "" + + # bdb-instance-name is the name of the BigchainDB instance in this cluster. bdb-instance-name: "" - # NGINX - ngx-instance-name: "" - # MongoDB Monitoring Agent + + # mdb-mon-instance-name is the name of the MongoDB Monitoring Agent instance + # in this cluster. mdb-mon-instance-name: "" - # MongoDB Backup Agent + + # mdb-bak-instance-name is the name of the MongoDB Backup Agent instance + # in this cluster. mdb-bak-instance-name: "" ---- -apiVersion: v1 -kind: ConfigMap -metadata: - name: mongodb-whitelist - namespace: default -data: - # We only support "all"" currently - allowed-hosts: "all" + + # ngx-mdb-instance-name is the FQDN of the MongoDB instance in this + # Kubernetes cluster. + ngx-mdb-instance-name: ".default.svc.cluster.local" + + # ngx-openresty-instance-name is the FQDN of the MongoDB instance in this + # Kubernetes cluster. + ngx-openresty-instance-name: ".default.svc.cluster.local" + + # ngx-bdb-instance-name is the FQDN of the BigchainDB instance in this + # Kubernetes cluster. + ngx-bdb-instance-name: ".default.svc.cluster.local" + + # mongodb-frontend-port is the port number on which external clients can + # access MongoDB. This needs to be restricted to only other MongoDB instances + # by enabling an authentication mechanism on MongoDB. + mongodb-frontend-port: "27017" + + # mongodb-backend-port is the port on which MongoDB is actually + # available/listening for requests. + mongodb-backend-port: "27017" + + # openresty-backend-port is the port number on which OpenResty is listening + # for requests. This is used by the NGINX instance to forward the requests to + # the right port, and by OpenResty instance to bind to the correct port to + # receive requests from NGINX instance. + openresty-backend-port: "80" + + # bigchaindb-api-port is the port number on which BigchainDB is listening + # for HTTP requests. + bigchaindb-api-port: "9984" + + # bigchaindb-ws-port is the port number on which BigchainDB is listening + # for Websocket requests. + bigchaindb-ws-port: "9985" + + # threescale-api-port is the port number on which 3scale can communicate with + # OpenResty backend. + threescale-api-port: "44433" + --- apiVersion: v1 kind: ConfigMap @@ -35,8 +91,10 @@ metadata: data: # Colon-separated list of all *other* nodes' BigchainDB public keys. bdb-keyring: "<':' separated list of public keys>" + # BigchainDB instance authentication user name bdb-user: "" + # BigchainDB public key of *this* node. # Example: "EPQk5i5yYpoUwGVM8VKZRjM8CYxB6j8Lu8i8SG7kGGce" bdb-public-key: "" diff --git a/k8s/configuration/secret.yaml b/k8s/configuration/secret.yaml index 26fb5b2f..eccaf2b7 100644 --- a/k8s/configuration/secret.yaml +++ b/k8s/configuration/secret.yaml @@ -109,9 +109,3 @@ data: service-id: "" version-header: "" provider-key: "" - # The frontend-api-dns-name will be DNS name registered for your HTTPS - # certificate. - frontend-api-dns-name: "" - # The upstream-api-port can be set to any port other than 9984, 9985, 443, - # 8888 and 27017. We usually use port '9999', which is 'OTk5OQo=' in base 64. - upstream-api-port: "OTk5OQo=" diff --git a/k8s/dev-setup/bigchaindb.yaml b/k8s/dev-setup/bigchaindb.yaml new file mode 100644 index 00000000..1186738e --- /dev/null +++ b/k8s/dev-setup/bigchaindb.yaml @@ -0,0 +1,95 @@ +apiVersion: v1 +kind: Service +metadata: + name: bdb + namespace: default + labels: + name: bdb +spec: + selector: + app: bdb-dep + ports: + - port: 9984 + targetPort: 9984 + name: bdb-api-port + protocol: TCP + - port: 9985 + targetPort: 9985 + name: bdb-ws-port + protocol: TCP + type: ClusterIP + clusterIP: None +--- +apiVersion: extensions/v1beta1 +kind: Deployment +metadata: + name: bdb-dep +spec: + replicas: 1 + template: + metadata: + labels: + app: bdb-dep + spec: + terminationGracePeriodSeconds: 10 + containers: + - name: bigchaindb + image: bigchaindb/bigchaindb:1.0.0 + imagePullPolicy: Always + args: + - start + env: + - name: BIGCHAINDB_DATABASE_HOST + value: mdb + - name: BIGCHAINDB_DATABASE_PORT + value: "27017" + - name: BIGCHAINDB_DATABASE_REPLICASET + value: bigchain-rs + - name: BIGCHAINDB_DATABASE_BACKEND + value: mongodb + - name: BIGCHAINDB_DATABASE_NAME + value: bigchain + - name: BIGCHAINDB_SERVER_BIND + value: "0.0.0.0:9984" + - name: BIGCHAINDB_WSSERVER_HOST + value: "0.0.0.0" + - name: BIGCHAINDB_WSSERVER_PORT + value: "9985" + - name: BIGCHAINDB_KEYPAIR_PUBLIC + value: "EEWUAhsk94ZUHhVw7qx9oZiXYDAWc9cRz93eMrsTG4kZ" + - name: BIGCHAINDB_KEYPAIR_PRIVATE + value: "3CjmRhu718gT1Wkba3LfdqX5pfYuBdaMPLd7ENUga5dm" + - name: BIGCHAINDB_BACKLOG_REASSIGN_DELAY + value: "120" + - name: BIGCHAINDB_DATABASE_MAXTRIES + value: "3" + - name: BIGCHAINDB_DATABASE_CONNECTION_TIMEOUT + value: "120" + - name: BIGCHAINDB_LOG_LEVEL_CONSOLE + value: debug + ports: + - containerPort: 9984 + hostPort: 9984 + name: bdb-port + protocol: TCP + - containerPort: 9985 + hostPort: 9985 + name: bdb-ws-port + protocol: TCP + resources: + limits: + cpu: 200m + memory: 768Mi + livenessProbe: + httpGet: + path: / + port: 9984 + initialDelaySeconds: 15 + timeoutSeconds: 10 + readinessProbe: + httpGet: + path: / + port: 9984 + initialDelaySeconds: 15 + timeoutSeconds: 10 + restartPolicy: Always diff --git a/k8s/dev-setup/mongo.yaml b/k8s/dev-setup/mongo.yaml new file mode 100644 index 00000000..c5c04d03 --- /dev/null +++ b/k8s/dev-setup/mongo.yaml @@ -0,0 +1,49 @@ +apiVersion: v1 +kind: Service +metadata: + name: mdb + namespace: default + labels: + name: mdb +spec: + selector: + app: mdb-ss + ports: + - port: 27017 + targetPort: 27017 + protocol: TCP + name: mdb-svc-port + type: ClusterIP + clusterIP: None +--- +apiVersion: apps/v1beta1 +kind: StatefulSet +metadata: + name: mdb-ss + namespace: default +spec: + serviceName: mdb + replicas: 1 + template: + metadata: + name: mdb-ss + labels: + app: mdb-ss + spec: + terminationGracePeriodSeconds: 10 + containers: + - name: mongodb + image: mongo:3.4.4 + imagePullPolicy: Always + args: + - --replSet + - bigchain-rs + ports: + - containerPort: 27017 + hostPort: 27017 + protocol: TCP + resources: + limits: + cpu: 200m + memory: 768Mi + restartPolicy: Always diff --git a/k8s/dev-setup/nginx-http.yaml b/k8s/dev-setup/nginx-http.yaml new file mode 100644 index 00000000..cf4bf260 --- /dev/null +++ b/k8s/dev-setup/nginx-http.yaml @@ -0,0 +1,79 @@ +apiVersion: v1 +kind: Service +metadata: + name: ngx-http + namespace: default + labels: + name: ngx-http + annotations: + # NOTE: the following annotation is a beta feature and + # only available in GCE/GKE and Azure as of now + # Ref: https://kubernetes.io/docs/tutorials/services/source-ip/ + service.beta.kubernetes.io/external-traffic: OnlyLocal +spec: + selector: + app: ngx-http-dep + ports: + - port: 80 + targetPort: 80 + name: ngx-public-bdb-port-http + protocol: TCP + - port: 8080 + targetPort: 8080 + name: public-health-check-port + protocol: TCP + - port: 27017 + targetPort: 27017 + name: ngx-public-mdb-port + protocol: TCP + type: LoadBalancer +--- +apiVersion: extensions/v1beta1 +kind: Deployment +metadata: + name: ngx-http-dep + namespace: default +spec: + replicas: 1 + template: + metadata: + name: ngx-http-dep + labels: + app: ngx-http-dep + spec: + terminationGracePeriodSeconds: 10 + containers: + - name: nginx-http + image: bigchaindb/nginx_http:1.0 + imagePullPolicy: Always + env: + - name: CLUSTER_FRONTEND_PORT + value: "80" + - name: HEALTH_CHECK_PORT + value: "8080" + - name: DNS_SERVER + value: "10.0.0.10" + - name: MONGODB_FRONTEND_PORT + value: "27017" + - name: MONGODB_BACKEND_HOST + value: "mdb.default.svc.cluster.local" + - name: MONGODB_BACKEND_PORT + value: "27017" + - name: BIGCHAINDB_BACKEND_HOST + value: "bdb.default.svc.cluster.local" + - name: BIGCHAINDB_API_PORT + value: "9984" + - name: BIGCHAINDB_WS_PORT + value: "9985" + ports: + - containerPort: 27017 + protocol: TCP + - containerPort: 8080 + protocol: TCP + - containerPort: 80 + protocol: TCP + resources: + limits: + cpu: 200m + memory: 768Mi + restartPolicy: Always diff --git a/k8s/dev-setup/nginx-https.yaml b/k8s/dev-setup/nginx-https.yaml new file mode 100644 index 00000000..4763624d --- /dev/null +++ b/k8s/dev-setup/nginx-https.yaml @@ -0,0 +1,127 @@ +apiVersion: v1 +kind: Secret +metadata: + name: https-certs + namespace: default +type: Opaque +data: + # Base64-encoded HTTPS private key + cert.key: "" + # Base64-encoded HTTPS certificate chain + # starting with your primary SSL cert (e.g. your_domain.crt) + # followed by all intermediate certs. + # If cert if from DigiCert, download "Best format for nginx". + cert.pem: "" + service-id: "" + version-header: "" + provider-key: "" +--- +apiVersion: v1 +kind: Service +metadata: + name: openresty + namespace: default + labels: + name: openresty + annotations: + service.beta.kubernetes.io/external-traffic: OnlyLocal +spec: + selector: + app: openresty-dep + ports: + - port: 80 + targetPort: 80 + name: openresty-svc-port + protocol: TCP + type: ClusterIP + clusterIP: None +--- +apiVersion: extensions/v1beta1 +kind: Deployment +metadata: + name: openresty-dep +spec: + replicas: 1 + template: + metadata: + labels: + app: openresty-dep + spec: + terminationGracePeriodSeconds: 10 + containers: + - name: nginx-openresty + image: bigchaindb/nginx_3scale:2.0 + imagePullPolicy: Always + env: + - name: DNS_SERVER + value: "10.0.0.10" + - name: OPENRESTY_FRONTEND_PORT + value: "80" + - name: BIGCHAINDB_BACKEND_HOST + value: "bdb.default.svc.cluster.local" + - name: BIGCHAINDB_API_PORT + value: "9984" + ports: + - containerPort: 80 + protocol: TCP + name: openresty-port + volumeMounts: + - name: threescale-credentials + mountPath: /usr/local/openresty/nginx/conf/threescale + readOnly: true + livenessProbe: + httpGet: + path: / + port: openresty-port + initialDelaySeconds: 15 + periodSeconds: 15 + failureThreshold: 3 + timeoutSeconds: 10 + resources: + limits: + cpu: 200m + memory: 768Mi + restartPolicy: Always + volumes: + - name: threescale-credentials + secret: + secretName: threescale-credentials + defaultMode: 0400 diff --git a/k8s/nginx-3scale/nginx-3scale-dep.yaml b/k8s/nginx-3scale/nginx-3scale-dep.yaml deleted file mode 100644 index 7951e14d..00000000 --- a/k8s/nginx-3scale/nginx-3scale-dep.yaml +++ /dev/null @@ -1,94 +0,0 @@ -############################################################### -# This config file runs nginx as a k8s deployment and exposes # -# it using an external load balancer. # -# This deployment is used as a front end to both BigchainDB # -# and MongoDB. # -############################################################### - -apiVersion: extensions/v1beta1 -kind: Deployment -metadata: - name: ngx-instance-0-dep -spec: - replicas: 1 - template: - metadata: - labels: - app: ngx-instance-0-dep - spec: - terminationGracePeriodSeconds: 10 - containers: - - name: nginx-3scale - image: bigchaindb/nginx_3scale:1.5 - imagePullPolicy: Always - env: - - name: MONGODB_FRONTEND_PORT - value: $(NGX_INSTANCE_0_SERVICE_PORT_NGX_PUBLIC_MDB_PORT) - - name: MONGODB_BACKEND_HOST - # NGINX requires FQDN to resolve names - value: mdb-instance-0.default.svc.cluster.local - - name: MONGODB_BACKEND_PORT - value: "27017" - - name: BIGCHAINDB_FRONTEND_PORT - value: $(NGX_INSTANCE_0_SERVICE_PORT_NGX_PUBLIC_BDB_PORT) - - name: BIGCHAINDB_BACKEND_HOST - value: bdb-instance-0.default.svc.cluster.local - - name: BIGCHAINDB_BACKEND_PORT - value: "9984" - - name: MONGODB_WHITELIST - valueFrom: - configMapKeyRef: - name: mongodb-whitelist - key: allowed-hosts - - name: DNS_SERVER - value: "10.0.0.10" - - name: NGINX_HEALTH_CHECK_PORT - value: "8888" - ports: - - containerPort: 27017 - hostPort: 27017 - name: public-mdb-port - protocol: TCP - - containerPort: 443 - hostPort: 443 - name: public-bdb-port - protocol: TCP - - containerPort: 80 - hostPort: 80 - name: https-msg-port - protocol: TCP - - containerPort: 8888 - hostPort: 8888 - name: health-check - protocol: TCP - - containerPort: 8080 - hostPort: 8080 - name: public-api-port - protocol: TCP - volumeMounts: - - name: threescale-credentials - mountPath: /usr/local/openresty/nginx/conf/threescale - readOnly: true - - name: https-certs - mountPath: /usr/local/openresty/nginx/conf/ssl/ - readOnly: true - resources: - limits: - cpu: 200m - memory: 768Mi - livenessProbe: - httpGet: - path: / - port: 8888 - initialDelaySeconds: 15 - timeoutSeconds: 10 - restartPolicy: Always - volumes: - - name: https-certs - secret: - secretName: https-certs - defaultMode: 0400 - - name: threescale-credentials - secret: - secretName: threescale-credentials - defaultMode: 0400 diff --git a/k8s/nginx-3scale/nginx-3scale-svc.yaml b/k8s/nginx-3scale/nginx-3scale-svc.yaml deleted file mode 100644 index 9150c24d..00000000 --- a/k8s/nginx-3scale/nginx-3scale-svc.yaml +++ /dev/null @@ -1,33 +0,0 @@ -apiVersion: v1 -kind: Service -metadata: - name: ngx-instance-0 - namespace: default - labels: - name: ngx-instance-0 - annotations: - # NOTE: the following annotation is a beta feature and - # only available in GCE/GKE and Azure as of now - # Ref: https://kubernetes.io/docs/tutorials/services/source-ip/ - service.beta.kubernetes.io/external-traffic: OnlyLocal -spec: - selector: - app: ngx-instance-0-dep - ports: - - port: 80 - targetPort: 80 - name: ngx-public-bdb-port-http - protocol: TCP - - port: 443 - targetPort: 443 - name: ngx-public-bdb-port - protocol: TCP - - port: 8080 - targetPort: 8080 - name: ngx-public-3scale-port - protocol: TCP - - port: 27017 - targetPort: 27017 - name: ngx-public-mdb-port - protocol: TCP - type: LoadBalancer diff --git a/k8s/nginx/container/Dockerfile b/k8s/nginx-http/container/Dockerfile similarity index 84% rename from k8s/nginx/container/Dockerfile rename to k8s/nginx-http/container/Dockerfile index 04c69e84..c6b0ccd4 100644 --- a/k8s/nginx/container/Dockerfile +++ b/k8s/nginx-http/container/Dockerfile @@ -1,4 +1,4 @@ -FROM nginx:1.11.10 +FROM nginx:1.13.1 LABEL maintainer "dev@bigchaindb.com" WORKDIR / RUN apt-get update \ @@ -7,5 +7,5 @@ RUN apt-get update \ && apt-get clean COPY nginx.conf.template /etc/nginx/nginx.conf COPY nginx_entrypoint.bash / -EXPOSE 80 81 443 444 27017 +EXPOSE 80 27017 ENTRYPOINT ["/nginx_entrypoint.bash"] diff --git a/k8s/nginx-http/container/README.md b/k8s/nginx-http/container/README.md new file mode 100644 index 00000000..860ffa4e --- /dev/null +++ b/k8s/nginx-http/container/README.md @@ -0,0 +1,15 @@ +## Nginx container for Secure WebSocket Support + + +### Step 1: Build and Push the Latest Container +Use the `docker_build_and_push.bash` script to build the latest docker image +and upload it to Docker Hub. +Ensure that the image tag is updated to a new version number to properly +reflect any changes made to the container. + + +### Note about testing Websocket connections: +You can test the WebSocket server by using +[wsc](https://www.npmjs.com/package/wsc) tool with a command like: + +`wsc -er ws://localhost:9985/api/v1/streams/valid_transactions`. diff --git a/k8s/nginx-http/container/docker_build_and_push.bash b/k8s/nginx-http/container/docker_build_and_push.bash new file mode 100755 index 00000000..d4b70555 --- /dev/null +++ b/k8s/nginx-http/container/docker_build_and_push.bash @@ -0,0 +1,5 @@ +#!/bin/bash + +docker build -t bigchaindb/nginx_http:1.0 . + +docker push bigchaindb/nginx_http:1.0 diff --git a/k8s/nginx-http/container/nginx.conf.template b/k8s/nginx-http/container/nginx.conf.template new file mode 100644 index 00000000..d2d90d26 --- /dev/null +++ b/k8s/nginx-http/container/nginx.conf.template @@ -0,0 +1,154 @@ +# Frontend API server that: +# 1. Forwards BDB HTTP & WS requests to BDB backend. +# 2. Forwards MDB TCP connections to MDB backend. +# 3. Does health check with LB (optional). + +worker_processes 2; +daemon off; +user nobody nogroup; +pid /tmp/nginx.pid; +error_log /dev/stderr; + +events { + # Each worker handles up to 512 connections. Increase this for heavy + # workloads. + worker_connections 512; + accept_mutex on; + use epoll; +} + +http { + access_log /dev/stdout combined buffer=16k flush=5s; + + # Allow 10 req/sec from the same IP address, and store the counters in a + # `zone` or shared memory location tagged as 'one'. + limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s; + + # Enable logging when requests are being throttled. + limit_req_log_level notice; + + # HTTP status code to return to the client when throttling; + # 429 is for TooManyRequests, ref. RFC 6585 + limit_req_status 429; + + # Limit requests from the same client, allow `burst` to 20 r/s, + # `nodelay` or drop connection immediately in case it exceeds this + # threshold. + limit_req zone=one burst=20 nodelay; + + # `slowloris` attack mitigation settings. + client_body_timeout 10s; + client_header_timeout 10s; + + # DNS resolver to use for all the backend names specified in this configuration. + resolver DNS_SERVER valid=30s ipv6=off; + + keepalive_timeout 60s; + + # The following map blocks enable lazy-binding to the backend at runtime, + # rather than binding as soon as NGINX starts. + map $remote_addr $bdb_backend { + default BIGCHAINDB_BACKEND_HOST; + } + + # Frontend server for the external clients + server { + listen CLUSTER_FRONTEND_PORT; + + underscores_in_headers on; + + # Forward websockets to backend BDB at 9985. + location /api/v1/streams/valid_transactions { + proxy_pass http://$bdb_backend:BIGCHAINDB_WS_PORT; + proxy_read_timeout 600s; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + } + + + # Forward other URL paths to backend BDB at 9984. + location / { + proxy_ignore_client_abort on; + proxy_set_header X-Real-IP $remote_addr; + + # max client request body size: avg transaction size. + client_max_body_size 15k; + + # No auth for GETs, forward directly to BDB. + if ($request_method = GET) { + proxy_pass http://$bdb_backend:BIGCHAINDB_API_PORT; + } + + # POST requests get forwarded to OpenResty instance. Enable CORS too. + if ($request_method = POST ) { + add_header 'Access-Control-Allow-Origin' '*'; + add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS'; + add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range'; + add_header 'Access-Control-Expose-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range'; + + proxy_pass http://$bdb_backend:BIGCHAINDB_API_PORT; + } + + # OPTIONS requests handling for CORS. + if ($request_method = 'OPTIONS') { + add_header 'Access-Control-Allow-Origin' '*'; + add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS'; + add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range,app_key,app_id'; + add_header 'Access-Control-Max-Age' 43200; + add_header 'Content-Type' 'text/plain charset=UTF-8'; + add_header 'Content-Length' 0; + return 204; + } + } + } + +# # Frontend server for the load balancer to respond to health checks. +# server { +# listen HEALTH_CHECK_PORT; +# +# location = /health { +# return 200; +# } +# } +} + +# NGINX stream block for TCP and UDP proxies. Used to proxy MDB TCP +# connection. +stream { + log_format mdb_log '[$time_iso8601] $realip_remote_addr $remote_addr ' + '$proxy_protocol_addr $proxy_protocol_port ' + '$protocol $status $session_time $bytes_sent ' + '$bytes_received "$upstream_addr" "$upstream_bytes_sent" ' + '"$upstream_bytes_received" "$upstream_connect_time" '; + + access_log /dev/stdout mdb_log buffer=16k flush=5s; + + # Define a zone 'two' of size 10 megabytes to store the counters + # that hold number of TCP connections from a specific IP address. + limit_conn_zone $binary_remote_addr zone=two:10m; + + # Enable logging when connections are being throttled. + limit_conn_log_level notice; + + # Allow 16 connections from the same IP address. + limit_conn two 16; + + # DNS resolver to use for all the backend names specified in this configuration. + resolver DNS_SERVER valid=30s ipv6=off; + + # The following map block enables lazy-binding to the backend at runtime, + # rather than binding as soon as NGINX starts. + map $remote_addr $mdb_backend { + default MONGODB_BACKEND_HOST; + } + + # Frontend server to forward connections to MDB instance. + server { + listen MONGODB_FRONTEND_PORT so_keepalive=10m:1m:5; + preread_timeout 30s; + tcp_nodelay on; + proxy_pass $mdb_backend:MONGODB_BACKEND_PORT; + } +} + diff --git a/k8s/nginx-http/container/nginx_entrypoint.bash b/k8s/nginx-http/container/nginx_entrypoint.bash new file mode 100755 index 00000000..5e379d08 --- /dev/null +++ b/k8s/nginx-http/container/nginx_entrypoint.bash @@ -0,0 +1,55 @@ +#!/bin/bash +set -euo pipefail + +# Cluster vars +cluster_frontend_port=`printenv CLUSTER_FRONTEND_PORT` + + +# NGINX vars +dns_server=`printenv DNS_SERVER` +health_check_port=`printenv HEALTH_CHECK_PORT` + + +# MongoDB vars +mongo_frontend_port=`printenv MONGODB_FRONTEND_PORT` +mongo_backend_host=`printenv MONGODB_BACKEND_HOST` +mongo_backend_port=`printenv MONGODB_BACKEND_PORT` + + +# BigchainDB vars +bdb_backend_host=`printenv BIGCHAINDB_BACKEND_HOST` +bdb_api_port=`printenv BIGCHAINDB_API_PORT` +bdb_ws_port=`printenv BIGCHAINDB_WS_PORT` + + +# sanity check +if [[ -z "${cluster_frontend_port}" || \ + -z "${mongo_frontend_port}" || \ + -z "${mongo_backend_host}" || \ + -z "${mongo_backend_port}" || \ + -z "${bdb_backend_host}" || \ + -z "${bdb_api_port}" || \ + -z "${bdb_ws_port}" || \ + -z "${dns_server}" || \ + -z "${health_check_port}" ]]; then + echo "Invalid environment settings detected. Exiting!" + exit 1 +fi + +NGINX_CONF_FILE=/etc/nginx/nginx.conf + +# configure the nginx.conf file with env variables +sed -i "s|CLUSTER_FRONTEND_PORT|${cluster_frontend_port}|g" ${NGINX_CONF_FILE} +sed -i "s|MONGODB_FRONTEND_PORT|${mongo_frontend_port}|g" ${NGINX_CONF_FILE} +sed -i "s|MONGODB_BACKEND_HOST|${mongo_backend_host}|g" ${NGINX_CONF_FILE} +sed -i "s|MONGODB_BACKEND_PORT|${mongo_backend_port}|g" ${NGINX_CONF_FILE} +sed -i "s|BIGCHAINDB_BACKEND_HOST|${bdb_backend_host}|g" ${NGINX_CONF_FILE} +sed -i "s|BIGCHAINDB_API_PORT|${bdb_api_port}|g" ${NGINX_CONF_FILE} +sed -i "s|BIGCHAINDB_WS_PORT|${bdb_ws_port}|g" ${NGINX_CONF_FILE} +sed -i "s|DNS_SERVER|${dns_server}|g" ${NGINX_CONF_FILE} +sed -i "s|HEALTH_CHECK_PORT|${health_check_port}|g" ${NGINX_CONF_FILE} + +# start nginx +echo "INFO: starting nginx..." +exec nginx -c /etc/nginx/nginx.conf + diff --git a/k8s/nginx-http/nginx-http-dep.yaml b/k8s/nginx-http/nginx-http-dep.yaml new file mode 100644 index 00000000..092b67fd --- /dev/null +++ b/k8s/nginx-http/nginx-http-dep.yaml @@ -0,0 +1,74 @@ +apiVersion: extensions/v1beta1 +kind: Deployment +metadata: + name: ngx-http-instance-0-dep +spec: + replicas: 1 + template: + metadata: + labels: + app: ngx-http-instance-0-dep + spec: + terminationGracePeriodSeconds: 10 + containers: + - name: nginx-http + image: bigchaindb/nginx_http:1.0 + imagePullPolicy: Always + env: + - name: CLUSTER_FRONTEND_PORT + valueFrom: + configMapKeyRef: + name: vars + key: cluster-frontend-port + - name: HEALTH_CHECK_PORT + valueFrom: + configMapKeyRef: + name: vars + key: cluster-health-check-port + - name: DNS_SERVER + valueFrom: + configMapKeyRef: + name: vars + key: cluster-dns-server-ip + - name: MONGODB_FRONTEND_PORT + valueFrom: + configMapKeyRef: + name: vars + key: mongodb-frontend-port + - name: MONGODB_BACKEND_HOST + valueFrom: + configMapKeyRef: + name: vars + key: ngx-mdb-instance-name + - name: MONGODB_BACKEND_PORT + valueFrom: + configMapKeyRef: + name: vars + key: mongodb-backend-port + - name: BIGCHAINDB_BACKEND_HOST + valueFrom: + configMapKeyRef: + name: vars + key: ngx-bdb-instance-name + - name: BIGCHAINDB_API_PORT + valueFrom: + configMapKeyRef: + name: vars + key: bigchaindb-api-port + - name: BIGCHAINDB_WS_PORT + valueFrom: + configMapKeyRef: + name: vars + key: bigchaindb-ws-port + ports: + - containerPort: "" + protocol: TCP + - containerPort: "" + protocol: TCP + - containerPort: "" + protocol: TCP + resources: + limits: + cpu: 200m + memory: 768Mi + restartPolicy: Always diff --git a/k8s/nginx-http/nginx-http-svc.yaml b/k8s/nginx-http/nginx-http-svc.yaml new file mode 100644 index 00000000..a655c958 --- /dev/null +++ b/k8s/nginx-http/nginx-http-svc.yaml @@ -0,0 +1,24 @@ +apiVersion: v1 +kind: Service +metadata: + name: ngx-http-instance-0 + namespace: default + labels: + name: ngx-http-instance-0 + annotations: + # NOTE: the following annotation is a beta feature and + # only available in GCE/GKE and Azure as of now + service.beta.kubernetes.io/external-traffic: OnlyLocal +spec: + selector: + app: ngx-http-instance-0-dep + ports: + - port: "" + targetPort: "" + name: public-cluster-port + protocol: TCP + - port: "" + targetPort: "" + name: public-health-check-port + protocol: TCP + type: LoadBalancer diff --git a/k8s/nginx-https/container/Dockerfile b/k8s/nginx-https/container/Dockerfile new file mode 100644 index 00000000..98ec0cfd --- /dev/null +++ b/k8s/nginx-https/container/Dockerfile @@ -0,0 +1,11 @@ +FROM nginx:1.13.1 +LABEL maintainer "dev@bigchaindb.com" +WORKDIR / +RUN apt-get update \ + && apt-get -y upgrade \ + && apt-get autoremove \ + && apt-get clean +COPY nginx.conf.template /etc/nginx/nginx.conf +COPY nginx_entrypoint.bash / +EXPOSE 80 443 27017 +ENTRYPOINT ["/nginx_entrypoint.bash"] diff --git a/k8s/nginx-https/container/README.md b/k8s/nginx-https/container/README.md new file mode 100644 index 00000000..1f649b10 --- /dev/null +++ b/k8s/nginx-https/container/README.md @@ -0,0 +1,15 @@ +## Nginx container for Secure WebSocket Support + + +### Step 1: Build and Push the Latest Container +Use the `docker_build_and_push.bash` script to build the latest docker image +and upload it to Docker Hub. +Ensure that the image tag is updated to a new version number to properly +reflect any changes made to the container. + + +### Note about testing Websocket connections: +You can test the WebSocket server by using +[wsc](https://www.npmjs.com/package/wsc) tool with a command like: + +`wsc -er wss://localhost:9985/api/v1/streams/valid_transactions`. diff --git a/k8s/nginx-https/container/docker_build_and_push.bash b/k8s/nginx-https/container/docker_build_and_push.bash new file mode 100755 index 00000000..3ae71ff9 --- /dev/null +++ b/k8s/nginx-https/container/docker_build_and_push.bash @@ -0,0 +1,5 @@ +#!/bin/bash + +docker build -t bigchaindb/nginx_https:1.0 . + +docker push bigchaindb/nginx_https:1.0 diff --git a/k8s/nginx-https/container/nginx.conf.template b/k8s/nginx-https/container/nginx.conf.template new file mode 100644 index 00000000..6a8ecbac --- /dev/null +++ b/k8s/nginx-https/container/nginx.conf.template @@ -0,0 +1,193 @@ +# Frontend API server that: +# 1. Acts as the HTTPS termination point. +# 2. Forwards BDB HTTP requests to OpenResty backend. +# 3. Forwards BDB WS requests to BDB backend. +# 4. Forwards MDB TCP connections to MDB backend. +# 5. Forwards requests from 3scale to OpenResty backend. +# 6. Does health check with LB. + +worker_processes 2; +daemon off; +user nobody nogroup; +pid /tmp/nginx.pid; +error_log /dev/stderr; + +events { + # Each worker handles up to 512 connections. Increase this for heavy + # workloads. + worker_connections 512; + accept_mutex on; + use epoll; +} + +http { + access_log /dev/stdout combined buffer=16k flush=5s; + + # Allow 10 req/sec from the same IP address, and store the counters in a + # `zone` or shared memory location tagged as 'one'. + limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s; + + # Enable logging when requests are being throttled. + limit_req_log_level notice; + + # HTTP status code that is returned to the client; 429 is for TooManyRequests, + # ref. RFC 6585 + limit_req_status 429; + + # Limit requests from the same client, allow `burst` to 20 r/s, + # `nodelay` or drop connection immediately in case it exceeds this + # threshold. + limit_req zone=one burst=20 nodelay; + + # `slowloris` attack mitigation settings. + client_body_timeout 10s; + client_header_timeout 10s; + + # DNS resolver to use for all the backend names specified in this configuration. + resolver DNS_SERVER valid=30s ipv6=off; + + keepalive_timeout 60s; + + # The following map blocks enable lazy-binding to the backend at runtime, + # rather than binding as soon as NGINX starts. + map $remote_addr $bdb_backend { + default BIGCHAINDB_BACKEND_HOST; + } + map $remote_addr $openresty_backend { + default OPENRESTY_BACKEND_HOST; + } + + # Frontend server for the external clients; acts as HTTPS termination point. + server { + listen CLUSTER_FRONTEND_PORT ssl; + server_name "CLUSTER_FQDN"; + ssl_certificate /etc/nginx/ssl/cert.pem; + ssl_certificate_key /etc/nginx/ssl/cert.key; + ssl_protocols TLSv1 TLSv1.1 TLSv1.2; + ssl_ciphers HIGH:!aNULL:!MD5; + + underscores_in_headers on; + + # Forward websockets directly to backend BDB. + location /api/v1/streams/valid_transactions { + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + + proxy_pass http://$bdb_backend:BIGCHAINDB_WS_PORT; + proxy_read_timeout 600s; + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + } + + # Forward other URL paths as per business logic/use case to BDB or + # OpenResty instance. + location / { + proxy_ignore_client_abort on; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + + # max client request body size: avg transaction size. + client_max_body_size 15k; + + # No auth for GETs, forward directly to BDB. + if ($request_method = GET) { + proxy_pass http://$bdb_backend:BIGCHAINDB_API_PORT; + } + + # POST requests get forwarded to OpenResty instance. Enable CORS too. + if ($request_method = POST ) { + add_header 'Access-Control-Allow-Origin' '*'; + add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS'; + add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range'; + add_header 'Access-Control-Expose-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range'; + + proxy_pass http://$openresty_backend:OPENRESTY_BACKEND_PORT; + } + + # OPTIONS requests handling for CORS. + if ($request_method = 'OPTIONS') { + add_header 'Access-Control-Allow-Origin' '*'; + add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS'; + add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range,app_key,app_id'; + add_header 'Access-Control-Max-Age' 43200; + add_header 'Content-Type' 'text/plain charset=UTF-8'; + add_header 'Content-Length' 0; + return 204; + } + } + } + + # Frontend server for the load balancer to respond to health checks. + server { + listen HEALTH_CHECK_PORT; + + location = /health { + return 200; + } + } + + # Frontend server for the external clients; returns a pretty error message + # when an HTTP request is sent instead of HTTPS. + server { + listen 80; + server_name "CLUSTER_FQDN"; + + location / { + add_header Upgrade "TLS/1.2, HTTP/1.1" always; + default_type text/plain; + return 426 'Consider using the HTTPS protocol next time!'; + } + } + + # Frontend server to respond to requests from 3scale; forward directly to + # OpenResty instance. + server { + sendfile on; + listen THREESCALE_API_PORT; + + location / { + proxy_pass http://$openresty_backend:OPENRESTY_BACKEND_PORT; + } + } +} + +# NGINX stream block for TCP and UDP proxies. Used to proxy MDB TCP +# connection. +stream { + log_format mdb_log '[$time_iso8601] $realip_remote_addr $remote_addr ' + '$proxy_protocol_addr $proxy_protocol_port ' + '$protocol $status $session_time $bytes_sent ' + '$bytes_received "$upstream_addr" "$upstream_bytes_sent" ' + '"$upstream_bytes_received" "$upstream_connect_time" '; + + access_log /dev/stdout mdb_log buffer=16k flush=5s; + + # Define a zone 'two' of size 10 megabytes to store the counters + # that hold number of TCP connections from a specific IP address. + limit_conn_zone $binary_remote_addr zone=two:10m; + + # Enable logging when connections are being throttled. + limit_conn_log_level notice; + + # Allow 16 connections from the same IP address. + limit_conn two 16; + + # DNS resolver to use for all the backend names specified in this configuration. + resolver DNS_SERVER valid=30s ipv6=off; + + # The following map block enables lazy-binding to the backend at runtime, + # rather than binding as soon as NGINX starts. + map $remote_addr $mdb_backend { + default MONGODB_BACKEND_HOST; + } + + # Frontend server to forward connections to MDB instance. + server { + listen MONGODB_FRONTEND_PORT so_keepalive=10m:1m:5; + preread_timeout 30s; + tcp_nodelay on; + proxy_pass $mdb_backend:MONGODB_BACKEND_PORT; + } +} + diff --git a/k8s/nginx-https/container/nginx_entrypoint.bash b/k8s/nginx-https/container/nginx_entrypoint.bash new file mode 100755 index 00000000..474be806 --- /dev/null +++ b/k8s/nginx-https/container/nginx_entrypoint.bash @@ -0,0 +1,70 @@ +#!/bin/bash +set -euo pipefail + +# Cluster vars +cluster_fqdn=`printenv CLUSTER_FQDN` +cluster_frontend_port=`printenv CLUSTER_FRONTEND_PORT` + + +# NGINX vars +dns_server=`printenv DNS_SERVER` +health_check_port=`printenv HEALTH_CHECK_PORT` + + +# MongoDB vars +mongo_frontend_port=`printenv MONGODB_FRONTEND_PORT` +mongo_backend_host=`printenv MONGODB_BACKEND_HOST` +mongo_backend_port=`printenv MONGODB_BACKEND_PORT` + + +# OpenResty vars +openresty_backend_host=`printenv OPENRESTY_BACKEND_HOST` +openresty_backend_port=`printenv OPENRESTY_BACKEND_PORT` +threescale_api_port=`printenv THREESCALE_API_PORT` + + +# BigchainDB vars +bdb_backend_host=`printenv BIGCHAINDB_BACKEND_HOST` +bdb_api_port=`printenv BIGCHAINDB_API_PORT` +bdb_ws_port=`printenv BIGCHAINDB_WS_PORT` + + +# sanity check +if [[ -z "${cluster_frontend_port}" || \ + -z "${mongo_frontend_port}" || \ + -z "${mongo_backend_host}" || \ + -z "${mongo_backend_port}" || \ + -z "${openresty_backend_port}" || \ + -z "${openresty_backend_host}" || \ + -z "${threescale_api_port}" || \ + -z "${bdb_backend_host}" || \ + -z "${bdb_api_port}" || \ + -z "${bdb_ws_port}" || \ + -z "${dns_server}" || \ + -z "${health_check_port}" || \ + -z "${cluster_fqdn}" ]]; then + echo "Invalid environment settings detected. Exiting!" + exit 1 +fi + +NGINX_CONF_FILE=/etc/nginx/nginx.conf + +# configure the nginx.conf file with env variables +sed -i "s|CLUSTER_FQDN|${cluster_fqdn}|g" ${NGINX_CONF_FILE} +sed -i "s|CLUSTER_FRONTEND_PORT|${cluster_frontend_port}|g" ${NGINX_CONF_FILE} +sed -i "s|MONGODB_FRONTEND_PORT|${mongo_frontend_port}|g" ${NGINX_CONF_FILE} +sed -i "s|MONGODB_BACKEND_HOST|${mongo_backend_host}|g" ${NGINX_CONF_FILE} +sed -i "s|MONGODB_BACKEND_PORT|${mongo_backend_port}|g" ${NGINX_CONF_FILE} +sed -i "s|OPENRESTY_BACKEND_PORT|${openresty_backend_port}|g" ${NGINX_CONF_FILE} +sed -i "s|OPENRESTY_BACKEND_HOST|${openresty_backend_host}|g" ${NGINX_CONF_FILE} +sed -i "s|THREESCALE_API_PORT|${threescale_api_port}|g" ${NGINX_CONF_FILE} +sed -i "s|BIGCHAINDB_BACKEND_HOST|${bdb_backend_host}|g" ${NGINX_CONF_FILE} +sed -i "s|BIGCHAINDB_API_PORT|${bdb_api_port}|g" ${NGINX_CONF_FILE} +sed -i "s|BIGCHAINDB_WS_PORT|${bdb_ws_port}|g" ${NGINX_CONF_FILE} +sed -i "s|DNS_SERVER|${dns_server}|g" ${NGINX_CONF_FILE} +sed -i "s|HEALTH_CHECK_PORT|${health_check_port}|g" ${NGINX_CONF_FILE} + +# start nginx +echo "INFO: starting nginx..." +exec nginx -c /etc/nginx/nginx.conf + diff --git a/k8s/nginx-https/nginx-https-dep.yaml b/k8s/nginx-https/nginx-https-dep.yaml new file mode 100644 index 00000000..d756a6f7 --- /dev/null +++ b/k8s/nginx-https/nginx-https-dep.yaml @@ -0,0 +1,118 @@ +apiVersion: extensions/v1beta1 +kind: Deployment +metadata: + name: ngx-https-instance-0-dep +spec: + replicas: 1 + template: + metadata: + labels: + app: ngx-https-instance-0-dep + spec: + terminationGracePeriodSeconds: 10 + containers: + - name: nginx-https + image: bigchaindb/nginx_https:1.0 + imagePullPolicy: Always + env: + - name: CLUSTER_FRONTEND_PORT + valueFrom: + configMapKeyRef: + name: vars + key: cluster-frontend-port + - name: HEALTH_CHECK_PORT + valueFrom: + configMapKeyRef: + name: vars + key: cluster-health-check-port + - name: CLUSTER_FQDN + valueFrom: + configMapKeyRef: + name: vars + key: cluster-fqdn + - name: DNS_SERVER + valueFrom: + configMapKeyRef: + name: vars + key: cluster-dns-server-ip + - name: MONGODB_FRONTEND_PORT + valueFrom: + configMapKeyRef: + name: vars + key: mongodb-frontend-port + - name: MONGODB_BACKEND_HOST + valueFrom: + configMapKeyRef: + name: vars + key: ngx-mdb-instance-name + - name: MONGODB_BACKEND_PORT + valueFrom: + configMapKeyRef: + name: vars + key: mongodb-backend-port + - name: OPENRESTY_BACKEND_PORT + valueFrom: + configMapKeyRef: + name: vars + key: openresty-backend-port + - name: OPENRESTY_BACKEND_HOST + valueFrom: + configMapKeyRef: + name: vars + key: openresty-backend-host + - name: THREESCALE_API_PORT + valueFrom: + configMapKeyRef: + name: vars + key: threescale-api-port + - name: BIGCHAINDB_BACKEND_HOST + valueFrom: + configMapKeyRef: + name: vars + key: ngx-bdb-instance-name + - name: BIGCHAINDB_API_PORT + valueFrom: + configMapKeyRef: + name: vars + key: bigchaindb-api-port + - name: BIGCHAINDB_WS_PORT + valueFrom: + configMapKeyRef: + name: vars + key: bigchaindb-ws-port + ports: + # return a pretty error message on port 80, since we are expecting + # HTTPS traffic. + - containerPort: 80 + protocol: TCP + - containerPort: "" + protocol: TCP + - containerPort: "" + protocol: TCP + - containerPort: "" + protocol: TCP + - containerPort: "" + protocol: TCP + name: ngx-port + livenessProbe: + httpGet: + path: /health + port: ngx-port + initialDelaySeconds: 15 + periodSeconds: 15 + failureThreshold: 3 + timeoutSeconds: 10 + resources: + limits: + cpu: 200m + memory: 768Mi + volumeMounts: + - name: https-certs + mountPath: /etc/nginx/ssl/ + readOnly: true + restartPolicy: Always + volumes: + - name: https-certs + secret: + secretName: https-certs + defaultMode: 0400 diff --git a/k8s/nginx-https/nginx-https-svc.yaml b/k8s/nginx-https/nginx-https-svc.yaml new file mode 100644 index 00000000..4eac8eaf --- /dev/null +++ b/k8s/nginx-https/nginx-https-svc.yaml @@ -0,0 +1,32 @@ +apiVersion: v1 +kind: Service +metadata: + name: ngx-https-instance-0 + namespace: default + labels: + name: ngx-https-instance-0 + annotations: + # NOTE: the following annotation is a beta feature and + # only available in GCE/GKE and Azure as of now + service.beta.kubernetes.io/external-traffic: OnlyLocal +spec: + selector: + app: ngx-https-instance-0-dep + ports: + - port: "" + targetPort: "" + name: public-secure-cluster-port + protocol: TCP + - port: "" + targetPort: "" + name: public-mdb-port + protocol: TCP + - port: "" + targetPort: "" + name: public-threescale-port + protocol: TCP + - port: 80 + targetPort: 80 + name: public-insecure-cluster-port + protocol: TCP + type: LoadBalancer diff --git a/k8s/nginx-openresty/nginx-openresty-dep.yaml b/k8s/nginx-openresty/nginx-openresty-dep.yaml new file mode 100644 index 00000000..2ad4b393 --- /dev/null +++ b/k8s/nginx-openresty/nginx-openresty-dep.yaml @@ -0,0 +1,64 @@ +apiVersion: extensions/v1beta1 +kind: Deployment +metadata: + name: openresty-instance-0-dep +spec: + replicas: 1 + template: + metadata: + labels: + app: openresty-instance-0-dep + spec: + terminationGracePeriodSeconds: 10 + containers: + - name: nginx-openresty + image: bigchaindb/nginx_3scale:2.0 + imagePullPolicy: Always + env: + - name: DNS_SERVER + valueFrom: + configMapKeyRef: + name: vars + key: cluster-dns-server-ip + - name: OPENRESTY_FRONTEND_PORT + valueFrom: + configMapKeyRef: + name: vars + key: openresty-backend-port + - name: BIGCHAINDB_BACKEND_HOST + valueFrom: + configMapKeyRef: + name: vars + key: ngx-bdb-instance-name + - name: BIGCHAINDB_API_PORT + valueFrom: + configMapKeyRef: + name: vars + key: bigchaindb-api-port + ports: + - containerPort: "" + protocol: TCP + name: openresty-port + volumeMounts: + - name: threescale-credentials + mountPath: /usr/local/openresty/nginx/conf/threescale + readOnly: true + livenessProbe: + httpGet: + path: / + port: openresty-port + initialDelaySeconds: 15 + periodSeconds: 15 + successThreshold: 1 + failureThreshold: 3 + timeoutSeconds: 10 + resources: + limits: + cpu: 200m + memory: 768Mi + restartPolicy: Always + volumes: + - name: threescale-credentials + secret: + secretName: threescale-credentials + defaultMode: 0400 diff --git a/k8s/nginx-openresty/nginx-openresty-svc.yaml b/k8s/nginx-openresty/nginx-openresty-svc.yaml new file mode 100644 index 00000000..7f4cec69 --- /dev/null +++ b/k8s/nginx-openresty/nginx-openresty-svc.yaml @@ -0,0 +1,22 @@ +apiVersion: v1 +kind: Service +metadata: + name: openresty-instance-0 + namespace: default + labels: + name: openresty-instance-0 + annotations: + # NOTE: the following annotation is a beta feature and + # only available in GCE/GKE and Azure as of now + # Ref: https://kubernetes.io/docs/tutorials/services/source-ip/ + service.beta.kubernetes.io/external-traffic: OnlyLocal +spec: + selector: + app: openresty-instance-0-dep + ports: + - port: "" + targetPort: "" + name: openresty-svc-port + protocol: TCP + type: ClusterIP + clusterIP: None diff --git a/k8s/nginx/container/README.md b/k8s/nginx/container/README.md deleted file mode 100644 index f8baaba2..00000000 --- a/k8s/nginx/container/README.md +++ /dev/null @@ -1,78 +0,0 @@ -## Custom Nginx container for a Node - -### Need - -* Since, BigchainDB and MongoDB both need to expose ports to the outside - world (inter and intra cluster), we need to have a basic DDoS mitigation - strategy to ensure that we can provide proper uptime and security these - core services. - -* We can have a proxy like nginx/haproxy in every node that listens to - global connections and applies cluster level entry policy. - -### Implementation -* For MongoDB cluster communication, we will use nginx with an environment - variable specifying a ":" separated list of IPs in the whitelist. This list - contains the IPs of exising instances in the MongoDB replica set so as to - allow connections from the whitelist and avoid a DDoS. - -* For BigchainDB connections, nginx needs to have rules to throttle - connections that are using resources over a threshold. - - -### Step 1: Build the Latest Container - -Run `docker build -t bigchaindb/nginx:1.0 .` from this folder. - -Optional: Upload container to Docker Hub: -`docker push bigchaindb/nginx:1.0` - -### Step 2: Run the Container - -Note that the whilelist IPs must be specified with the subnet in the CIDR -format, eg: `1.2.3.4/16` - -``` -docker run \ - --env "MONGODB_FRONTEND_PORT=" \ - --env "MONGODB_BACKEND_HOST=" \ - --env "MONGODB_BACKEND_PORT=" \ - --env "BIGCHAINDB_FRONTEND_PORT=" \ - --env "BIGCHAINDB_BACKEND_HOST=" \ - --env "BIGCHAINDB_BACKEND_PORT=" \ - --env "BIGCHAINDB_WS_BACKEND_PORT=" \ - --env "BIGCHAINDB_WS_FRONTEND_PORT=" \ - --env "MONGODB_WHITELIST=" \ - --env "DNS_SERVER=" \ - --name=ngx \ - --publish=: \ - --publish=: \ - --rm=true \ - bigchaindb/nginx:1.0 -``` - -For example: -``` -docker run \ - --env="MONGODB_FRONTEND_PORT=17017" \ - --env="MONGODB_BACKEND_HOST=localhost" \ - --env="MONGODB_BACKEND_PORT=27017" \ - --env="BIGCHAINDB_FRONTEND_PORT=80" \ - --env="BIGCHAINDB_BACKEND_HOST=localhost" \ - --env="BIGCHAINDB_BACKEND_PORT=9984" \ - --env="BIGCHAINDB_WS_FRONTEND_PORT=81" \ - --env="BIGCHAINDB_WS_BACKEND_PORT=9985" \ - --env="MONGODB_WHITELIST=192.168.0.0/16:10.0.2.0/24" \ - --env="DNS_SERVER=127.0.0.1" \ - --name=ngx \ - --publish=80:80 \ - --publish=17017:17017 \ - --rm=true \ - bigchaindb/nginx:1.0 -``` - -### Note: -You can test the WebSocket server by using -[wsc](https://slack-redir.net/link?url=https%3A%2F%2Fwww.npmjs.com%2Fpackage%2Fwsc) tool with a command like: -`wsc -er ws://localhost:9985/api/v1/streams/valid_tx`. - diff --git a/k8s/nginx/container/nginx.conf.template b/k8s/nginx/container/nginx.conf.template deleted file mode 100644 index bae805a5..00000000 --- a/k8s/nginx/container/nginx.conf.template +++ /dev/null @@ -1,127 +0,0 @@ -worker_processes 2; -daemon off; -user nobody nogroup; -pid /tmp/nginx.pid; -error_log /etc/nginx/nginx.error.log; - -events { - worker_connections 256; - accept_mutex on; - use epoll; -} - -http { - server_names_hash_bucket_size 128; - access_log /etc/nginx/nginx.access.log combined buffer=16k flush=5s; - - # allow 10 req/sec from the same IP address, and store the counters in a - # `zone` or shared memory location tagged as 'one'. - limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s; - - # enable logging when requests are being throttled - limit_req_log_level notice; - - # the http status code to return to the client when throttling; - # 429 is for TooManyRequests, - # ref. RFC 6585 - limit_req_status 429; - - resolver DNS_SERVER valid=20s; - - map $remote_addr $bdb_backend { - default BIGCHAINDB_BACKEND_HOST; - } - - server { - listen BIGCHAINDB_FRONTEND_PORT; - # server_name "FRONTEND_DNS_NAME"; - underscores_in_headers on; - - # max client request body size: avg transaction size - client_max_body_size 15k; - - # keepalive connection settings - keepalive_timeout 20s; - - # `slowloris` attack mitigation settings - client_body_timeout 10s; - client_header_timeout 10s; - - location / { - proxy_ignore_client_abort on; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header Host $http_host; - proxy_redirect off; - - # TODO proxy_set_header X-Forwarded-Proto https; - - # limit requests from the same client, allow `burst` to 20 r/s, - # `nodelay` or drop connection immediately in case it exceeds this - # threshold. - limit_req zone=one burst=20 nodelay; - - proxy_pass http://$bdb_backend:BIGCHAINDB_BACKEND_PORT; - } - } - - server { - listen BIGCHAINDB_WS_FRONTEND_PORT; - underscores_in_headers on; - - # keepalive connection settings - keepalive_timeout 20s; - - # `slowloris` attack mitigation settings - client_body_timeout 10s; - client_header_timeout 10s; - - location /api/v1/streams/ { - proxy_read_timeout 300s; - proxy_pass http://$bdb_backend:BIGCHAINDB_WS_BACKEND_PORT; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection "upgrade"; - } - } -} - -# NGINX stream block for TCP and UDP proxies -stream { - log_format mdb_log '[$time_iso8601] $realip_remote_addr $remote_addr ' - '$proxy_protocol_addr $proxy_protocol_port ' - '$protocol $status $session_time $bytes_sent ' - '$bytes_received "$upstream_addr" "$upstream_bytes_sent" ' - '"$upstream_bytes_received" "$upstream_connect_time" '; - - access_log /etc/nginx/nginx.stream.access.log mdb_log buffer=16k flush=5s; - - # define a zone 'two' of size 10 megabytes to store the counters - # that hold number of TCP connections from a specific IP address - limit_conn_zone $binary_remote_addr zone=two:10m; - - # enable logging when connections are being throttled - limit_conn_log_level notice; - - resolver DNS_SERVER valid=20s; - - map $remote_addr $mdb_backend { - default MONGODB_BACKEND_HOST; - } - - server { - listen MONGODB_FRONTEND_PORT so_keepalive=10m:1m:5; - preread_timeout 30s; - tcp_nodelay on; - - # whitelist - #MONGODB_WHITELIST - allow all; - # deny access to everyone else - deny all; - - # allow 16 connections from the same IP address - limit_conn two 16; - - proxy_pass $mdb_backend:MONGODB_BACKEND_PORT; - } -} diff --git a/k8s/nginx/container/nginx_entrypoint.bash b/k8s/nginx/container/nginx_entrypoint.bash deleted file mode 100755 index 49578433..00000000 --- a/k8s/nginx/container/nginx_entrypoint.bash +++ /dev/null @@ -1,53 +0,0 @@ -#!/bin/bash -set -euo pipefail - -mongo_frontend_port=`printenv MONGODB_FRONTEND_PORT` -mongo_backend_host=`printenv MONGODB_BACKEND_HOST` -mongo_backend_port=`printenv MONGODB_BACKEND_PORT` -bdb_frontend_port=`printenv BIGCHAINDB_FRONTEND_PORT` -bdb_backend_host=`printenv BIGCHAINDB_BACKEND_HOST` -bdb_backend_port=`printenv BIGCHAINDB_BACKEND_PORT` -bdb_ws_frontend_port=`printenv BIGCHAINDB_WS_FRONTEND_PORT` -bdb_ws_backend_port=`printenv BIGCHAINDB_WS_BACKEND_PORT` -mongo_whitelist=`printenv MONGODB_WHITELIST` -dns_server=`printenv DNS_SERVER` - -# sanity checks -if [[ -z "${mongo_frontend_port}" || \ - -z "${mongo_backend_host}" || \ - -z "${mongo_backend_port}" || \ - -z "${bdb_frontend_port}" || \ - -z "${bdb_backend_host}" || \ - -z "${bdb_backend_port}" || \ - -z "${bdb_ws_backend_port}" || \ - -z "${bdb_ws_frontend_port}" || \ - -z "${dns_server}" ]] ; then - echo "Invalid environment settings detected. Exiting!" - exit 1 -fi - -NGINX_CONF_FILE=/etc/nginx/nginx.conf - -# configure the nginx.conf file with env variables -sed -i "s|MONGODB_FRONTEND_PORT|${mongo_frontend_port}|g" $NGINX_CONF_FILE -sed -i "s|MONGODB_BACKEND_HOST|${mongo_backend_host}|g" $NGINX_CONF_FILE -sed -i "s|MONGODB_BACKEND_PORT|${mongo_backend_port}|g" $NGINX_CONF_FILE -sed -i "s|BIGCHAINDB_FRONTEND_PORT|${bdb_frontend_port}|g" $NGINX_CONF_FILE -sed -i "s|BIGCHAINDB_BACKEND_HOST|${bdb_backend_host}|g" $NGINX_CONF_FILE -sed -i "s|BIGCHAINDB_BACKEND_PORT|${bdb_backend_port}|g" $NGINX_CONF_FILE -sed -i "s|BIGCHAINDB_WS_FRONTEND_PORT|${bdb_ws_frontend_port}|g" $NGINX_CONF_FILE -sed -i "s|BIGCHAINDB_WS_BACKEND_PORT|${bdb_ws_backend_port}|g" $NGINX_CONF_FILE -sed -i "s|DNS_SERVER|${dns_server}|g" $NGINX_CONF_FILE - -# populate the whitelist in the conf file as per MONGODB_WHITELIST env var -hosts=$(echo ${mongo_whitelist} | tr ":" "\n") -for host in $hosts; do - sed -i "s|MONGODB_WHITELIST|allow ${host};\n MONGODB_WHITELIST|g" $NGINX_CONF_FILE -done - -# remove the MONGODB_WHITELIST marker string from template -sed -i "s|MONGODB_WHITELIST||g" $NGINX_CONF_FILE - -# start nginx -echo "INFO: starting nginx..." -exec nginx -c /etc/nginx/nginx.conf diff --git a/k8s/nginx/nginx-dep.yaml b/k8s/nginx/nginx-dep.yaml deleted file mode 100644 index 133e2688..00000000 --- a/k8s/nginx/nginx-dep.yaml +++ /dev/null @@ -1,67 +0,0 @@ -############################################################### -# This config file runs nginx as a k8s deployment and exposes # -# it using an external load balancer. # -# This deployment is used as a front end to both BigchainDB # -# and MongoDB. # -############################################################### - -apiVersion: extensions/v1beta1 -kind: Deployment -metadata: - name: ngx-instance-0-dep -spec: - replicas: 1 - template: - metadata: - labels: - app: ngx-instance-0-dep - spec: - terminationGracePeriodSeconds: 10 - containers: - - name: nginx - image: bigchaindb/nginx:1.0 - imagePullPolicy: Always - env: - - name: MONGODB_FRONTEND_PORT - value: "27017" - - name: MONGODB_BACKEND_HOST - value: mdb-instance-0.default.svc.cluster.local - - name: MONGODB_BACKEND_PORT - value: "27017" - - name: BIGCHAINDB_FRONTEND_PORT - value: "80" - - name: BIGCHAINDB_BACKEND_HOST - value: bdb-instance-0.default.svc.cluster.local - - name: BIGCHAINDB_BACKEND_PORT - value: "9984" - - name: BIGCHAINDB_WS_FRONTEND_PORT - value: "81" - - name: BIGCHAINDB_WS_BACKEND_PORT - value: "9985" - - name: DNS_SERVER - value: "10.0.0.10" - - name: MONGODB_WHITELIST - valueFrom: - configMapKeyRef: - name: mongodb-whitelist - key: allowed-hosts - ports: - - containerPort: 27017 - hostPort: 27017 - name: public-mdb-port - protocol: TCP - - containerPort: 80 - hostPort: 80 - name: public-bdb-port - protocol: TCP - - containerPort: 81 - hostPort: 81 - name: public-ws-port - protocol: TCP - resources: - limits: - cpu: 200m - memory: 768Mi - #livenessProbe: TODO(Krish) - #readinessProbe: TODO(Krish) - restartPolicy: Always diff --git a/k8s/nginx/nginx-svc.yaml b/k8s/nginx/nginx-svc.yaml deleted file mode 100644 index 1e1e131c..00000000 --- a/k8s/nginx/nginx-svc.yaml +++ /dev/null @@ -1,28 +0,0 @@ -apiVersion: v1 -kind: Service -metadata: - name: ngx-instance-0 - namespace: default - labels: - name: ngx-instance-0 - annotations: - # NOTE: the following annotation is a beta feature and - # only available in GCE/GKE and Azure as of now - service.beta.kubernetes.io/external-traffic: OnlyLocal -spec: - selector: - app: ngx-instance-0-dep - ports: - - port: 27017 - targetPort: 27017 - name: ngx-public-mdb-port - protocol: TCP - - port: 80 - targetPort: 80 - name: ngx-public-api-port - protocol: TCP - - port: 81 - targetPort: 81 - name: ngx-public-ws-port - protocol: TCP - type: LoadBalancer