mirror of
https://github.com/bigchaindb/bigchaindb.git
synced 2024-10-13 13:34:05 +00:00
315 lines
10 KiB
ReStructuredText
315 lines
10 KiB
ReStructuredText
Log Analytics on Azure
|
|
======================
|
|
|
|
This section documents how to create and configure a Log Analytics workspace on
|
|
Azure, for a Kubernetes-based deployment.
|
|
The documented approach is based on an integration of Microsoft's Operations
|
|
Management Suite (OMS) with a Kubernetes-based Azure Container Service cluster.
|
|
|
|
The :ref:`oms-k8s-references` section (below) contains links to more detailed documentation on
|
|
Azure, and Kubernetes.
|
|
|
|
There are three main steps involved:
|
|
|
|
1. Create a workspace (``LogAnalyticsOMS``).
|
|
2. Create a ``ContainersOMS`` solution under the workspace.
|
|
3. Deploy the OMS agent(s).
|
|
|
|
Steps 1 and 2 rely on `Azure Resource Manager templates`_ and can be done with
|
|
one template so we'll cover them together. Step 3 relies on a
|
|
`Kubernetes DaemonSet`_ and will be covered separately.
|
|
|
|
Minimum Requirements
|
|
--------------------
|
|
This document assumes that you have already deployed a Kubernetes cluster, and
|
|
that you have the Kubernetes command line interface ``kubectl`` installed.
|
|
|
|
Creating a Workspace and Adding a Containers Solution
|
|
-----------------------------------------------------
|
|
For the sake of this document and example, we'll assume an existing resource
|
|
group named:
|
|
|
|
* ``resource_group``
|
|
|
|
and the workspace we'll create will be named:
|
|
|
|
* ``work_space``
|
|
|
|
If you feel creative you may replace these names by more interesting ones.
|
|
|
|
.. code-block:: bash
|
|
|
|
$ az group deployment create --debug \
|
|
--resource-group resource_group \
|
|
--name "Microsoft.LogAnalyticsOMS" \
|
|
--template-file log_analytics_oms.json \
|
|
--parameters @log_analytics_oms.parameters.json
|
|
|
|
An example of a simple template file (``--template-file``):
|
|
|
|
.. code-block:: json
|
|
|
|
{
|
|
"$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
|
|
"contentVersion": "1.0.0.0",
|
|
"parameters": {
|
|
"sku": {
|
|
"type": "String"
|
|
},
|
|
"workspaceName": {
|
|
"type": "String"
|
|
},
|
|
"solutionType": {
|
|
"type": "String"
|
|
},
|
|
},
|
|
"resources": [
|
|
{
|
|
"apiVersion": "2015-03-20",
|
|
"type": "Microsoft.OperationalInsights/workspaces",
|
|
"name": "[parameters('workspaceName')]",
|
|
"location": "[resourceGroup().location]",
|
|
"properties": {
|
|
"sku": {
|
|
"name": "[parameters('sku')]"
|
|
}
|
|
},
|
|
"resources": [
|
|
{
|
|
"apiVersion": "2015-11-01-preview",
|
|
"location": "[resourceGroup().location]",
|
|
"name": "[Concat(parameters('solutionType'), '(', parameters('workspaceName'), ')')]",
|
|
"type": "Microsoft.OperationsManagement/solutions",
|
|
"id": "[Concat(resourceGroup().id, '/providers/Microsoft.OperationsManagement/solutions/', parameters('solutionType'), '(', parameters('workspaceName'), ')')]",
|
|
"dependsOn": [
|
|
"[concat('Microsoft.OperationalInsights/workspaces/', parameters('workspaceName'))]"
|
|
],
|
|
"properties": {
|
|
"workspaceResourceId": "[resourceId('Microsoft.OperationalInsights/workspaces/', parameters('workspaceName'))]"
|
|
},
|
|
"plan": {
|
|
"publisher": "Microsoft",
|
|
"product": "[Concat('OMSGallery/', parameters('solutionType'))]",
|
|
"name": "[Concat(parameters('solutionType'), '(', parameters('workspaceName'), ')')]",
|
|
"promotionCode": ""
|
|
}
|
|
}
|
|
]
|
|
}
|
|
]
|
|
}
|
|
|
|
An example of the associated parameter file (``--parameters``):
|
|
|
|
.. code-block:: json
|
|
|
|
{
|
|
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
|
|
"contentVersion": "1.0.0.0",
|
|
"parameters": {
|
|
"sku": {
|
|
"value": "Free"
|
|
},
|
|
"workspaceName": {
|
|
"value": "work_space"
|
|
},
|
|
"solutionType": {
|
|
"value": "Containers"
|
|
},
|
|
}
|
|
}
|
|
|
|
Deploy the OMS Agents
|
|
---------------------
|
|
To deploy an OMS agent, two important pieces of information are needed:
|
|
|
|
* workspace id
|
|
* workspace key
|
|
|
|
You can obtain the workspace id using:
|
|
|
|
.. code-block:: bash
|
|
|
|
$ az resource show \
|
|
--resource-group resource_group
|
|
--resource-type Microsoft.OperationalInsights/workspaces
|
|
--name work_space \
|
|
| grep customerId
|
|
"customerId": "12345678-1234-1234-1234-123456789012",
|
|
|
|
Until we figure out a way to obtain the *workspace key* via the command line,
|
|
you can get it via the OMS Portal.
|
|
To get to the OMS Portal, go to the Azure Portal and click on:
|
|
|
|
Resource Groups > (Your k8s cluster's resource group) > Log analytics (OMS) > (Name of the only item listed) > OMS Workspace > OMS Portal
|
|
|
|
(Let us know if you find a faster way.)
|
|
Then see `Microsoft's instructions to obtain your workspace ID and key
|
|
<https://docs.microsoft.com/en-us/azure/container-service/container-service-kubernetes-oms#obtain-your-workspace-id-and-key>`_ (via the OMS Portal).
|
|
|
|
Once you have the workspace id and key, you can include them in the following
|
|
YAML file (:download:`oms-daemonset.yaml
|
|
<../../../../k8s/logging-and-monitoring/oms-daemonset.yaml>`):
|
|
|
|
.. code-block:: yaml
|
|
|
|
# oms-daemonset.yaml
|
|
apiVersion: extensions/v1beta1
|
|
kind: DaemonSet
|
|
metadata:
|
|
name: omsagent
|
|
spec:
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: omsagent
|
|
spec:
|
|
containers:
|
|
- env:
|
|
- name: WSID
|
|
value: <workspace_id>
|
|
- name: KEY
|
|
value: <workspace_key>
|
|
image: microsoft/oms
|
|
name: omsagent
|
|
ports:
|
|
- containerPort: 25225
|
|
protocol: TCP
|
|
securityContext:
|
|
privileged: true
|
|
volumeMounts:
|
|
- mountPath: /var/run/docker.sock
|
|
name: docker-sock
|
|
volumes:
|
|
- name: docker-sock
|
|
hostPath:
|
|
path: /var/run/docker.sock
|
|
|
|
To deploy the OMS agents (one per Kubernetes node, i.e. one per computer),
|
|
simply run the following command:
|
|
|
|
.. code-block:: bash
|
|
|
|
$ kubectl create -f oms-daemonset.yaml
|
|
|
|
|
|
Search the OMS Logs
|
|
-------------------
|
|
|
|
OMS should now be getting, storing and indexing all the logs
|
|
from all the containers in your Kubernetes cluster.
|
|
You can search the OMS logs from the Azure Portal
|
|
or the OMS Portal, but at the time of writing,
|
|
there was more functionality in the OMS Portal
|
|
(e.g. the ability to create an Alert based on a search).
|
|
|
|
There are instructions to get to the OMS Portal
|
|
in the section titled :ref:`Deploy the OMS Agents` above.
|
|
Once you're in the OMS Portal, click on **Log Search**
|
|
and enter a query.
|
|
Here are some example queries:
|
|
|
|
All logging messages containing the strings "critical" or "error" (not case-sensitive):
|
|
|
|
``Type=ContainerLog (critical OR error)``
|
|
|
|
.. note::
|
|
|
|
You can filter the results even more by clicking on things in the left sidebar.
|
|
For OMS Log Search syntax help, see the
|
|
`Log Analytics search reference <https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-search-reference>`_.
|
|
|
|
All logging messages containing the string "error" but not "404":
|
|
|
|
``Type=ContainerLog error NOT(404)``
|
|
|
|
All logging messages containing the string "critical" but not "CriticalAddonsOnly":
|
|
|
|
``Type=ContainerLog critical NOT(CriticalAddonsOnly)``
|
|
|
|
All logging messages from containers running the Docker image bigchaindb/nginx_3scale:1.3, containing the string "GET" but not the strings "Go-http-client" or "runscope" (where those exclusions filter out tests by Kubernetes and Runscope):
|
|
|
|
``Type=ContainerLog Image="bigchaindb/nginx_3scale:1.3" GET NOT("Go-http-client") NOT(runscope)``
|
|
|
|
.. note::
|
|
|
|
We wrote a small Python 3 script to analyze the logs found by the above NGINX search.
|
|
It's in ``k8s/logging-and-monitoring/analyze.py``. The docsting at the top
|
|
of the script explains how to use it.
|
|
|
|
|
|
Create an Email Alert
|
|
---------------------
|
|
|
|
Once you're satisfied with an OMS Log Search query string,
|
|
click the **🔔 Alert** icon in the top menu,
|
|
fill in the form,
|
|
and click **Save** when you're done.
|
|
|
|
|
|
Some Useful Management Tasks
|
|
----------------------------
|
|
List workspaces:
|
|
|
|
.. code-block:: bash
|
|
|
|
$ az resource list \
|
|
--resource-group resource_group \
|
|
--resource-type Microsoft.OperationalInsights/workspaces
|
|
|
|
List solutions:
|
|
|
|
.. code-block:: bash
|
|
|
|
$ az resource list \
|
|
--resource-group resource_group \
|
|
--resource-type Microsoft.OperationsManagement/solutions
|
|
|
|
Delete the containers solution:
|
|
|
|
.. code-block:: bash
|
|
|
|
$ az group deployment delete --debug \
|
|
--resource-group resource_group \
|
|
--name Microsoft.ContainersOMS
|
|
|
|
.. code-block:: bash
|
|
|
|
$ az resource delete \
|
|
--resource-group resource_group \
|
|
--resource-type Microsoft.OperationsManagement/solutions \
|
|
--name "Containers(work_space)"
|
|
|
|
Delete the workspace:
|
|
|
|
.. code-block:: bash
|
|
|
|
$ az group deployment delete --debug \
|
|
--resource-group resource_group \
|
|
--name Microsoft.LogAnalyticsOMS
|
|
|
|
.. code-block:: bash
|
|
|
|
$ az resource delete \
|
|
--resource-group resource_group \
|
|
--resource-type Microsoft.OperationalInsights/workspaces \
|
|
--name work_space
|
|
|
|
|
|
.. _oms-k8s-references:
|
|
|
|
References
|
|
----------
|
|
|
|
* `Monitor an Azure Container Service cluster with Microsoft Operations Management Suite (OMS) <https://docs.microsoft.com/en-us/azure/container-service/container-service-kubernetes-oms>`_
|
|
* `Manage Log Analytics using Azure Resource Manager templates <https://docs.microsoft.com/en-us/azure/log-analytics/log-analytics-template-workspace-configuration>`_
|
|
* `azure commands for deployments <https://docs.microsoft.com/en-us/cli/azure/group/deployment>`_
|
|
(``az group deployment``)
|
|
* `Understand the structure and syntax of Azure Resource Manager templates <https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-authoring-templates>`_
|
|
* `Kubernetes DaemonSet`_
|
|
|
|
|
|
|
|
.. _Azure Resource Manager templates: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-authoring-templates
|
|
.. _Kubernetes DaemonSet: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
|