Log Analytics on Azure ====================== This section documents how to create and configure a Log Analytics workspace on Azure, for a Kubernetes-based deployment. The documented approach is based on an integration of Microsoft's Operations Management Suite (OMS) with a Kubernetes-based Azure Container Service cluster. The :ref:`oms-k8s-references` section (below) contains links to more detailed documentation on Azure, and Kubernetes. There are three main steps involved: 1. Create a workspace (``LogAnalyticsOMS``). 2. Create a ``ContainersOMS`` solution under the workspace. 3. Deploy the OMS agent(s). Steps 1 and 2 rely on `Azure Resource Manager templates`_ and can be done with one template so we'll cover them together. Step 3 relies on a `Kubernetes DaemonSet`_ and will be covered separately. Minimum Requirements -------------------- This document assumes that you have already deployed a Kubernetes cluster, and that you have the Kubernetes command line interface ``kubectl`` installed. Creating a Workspace and Adding a Containers Solution ----------------------------------------------------- For the sake of this document and example, we'll assume an existing resource group named: * ``resource_group`` and the workspace we'll create will be named: * ``work_space`` If you feel creative you may replace these names by more interesting ones. .. code-block:: bash $ az group deployment create --debug \ --resource-group resource_group \ --name "Microsoft.LogAnalyticsOMS" \ --template-file log_analytics_oms.json \ --parameters @log_analytics_oms.parameters.json An example of a simple template file (``--template-file``): .. code-block:: json { "$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#", "contentVersion": "1.0.0.0", "parameters": { "sku": { "type": "String" }, "workspaceName": { "type": "String" }, "solutionType": { "type": "String" }, }, "resources": [ { "apiVersion": "2015-03-20", "type": "Microsoft.OperationalInsights/workspaces", "name": "[parameters('workspaceName')]", "location": "[resourceGroup().location]", "properties": { "sku": { "name": "[parameters('sku')]" } }, "resources": [ { "apiVersion": "2015-11-01-preview", "location": "[resourceGroup().location]", "name": "[Concat(parameters('solutionType'), '(', parameters('workspaceName'), ')')]", "type": "Microsoft.OperationsManagement/solutions", "id": "[Concat(resourceGroup().id, '/providers/Microsoft.OperationsManagement/solutions/', parameters('solutionType'), '(', parameters('workspaceName'), ')')]", "dependsOn": [ "[concat('Microsoft.OperationalInsights/workspaces/', parameters('workspaceName'))]" ], "properties": { "workspaceResourceId": "[resourceId('Microsoft.OperationalInsights/workspaces/', parameters('workspaceName'))]" }, "plan": { "publisher": "Microsoft", "product": "[Concat('OMSGallery/', parameters('solutionType'))]", "name": "[Concat(parameters('solutionType'), '(', parameters('workspaceName'), ')')]", "promotionCode": "" } } ] } ] } An example of the associated parameter file (``--parameters``): .. code-block:: json { "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#", "contentVersion": "1.0.0.0", "parameters": { "sku": { "value": "Free" }, "workspaceName": { "value": "work_space" }, "solutionType": { "value": "Containers" }, } } Deploy the OMS Agents --------------------- To deploy an OMS agent, two important pieces of information are needed: * workspace id * workspace key You can obtain the workspace id using: .. code-block:: bash $ az resource show \ --resource-group resource_group --resource-type Microsoft.OperationalInsights/workspaces --name work_space \ | grep customerId "customerId": "12345678-1234-1234-1234-123456789012", Until we figure out a way to obtain the *workspace key* via the command line, you can get it via the OMS Portal. To get to the OMS Portal, go to the Azure Portal and click on: Resource Groups > (Your k8s cluster's resource group) > Log analytics (OMS) > (Name of the only item listed) > OMS Workspace > OMS Portal (Let us know if you find a faster way.) Then see `Microsoft's instructions to obtain your workspace ID and key `_ (via the OMS Portal). Once you have the workspace id and key, you can include them in the following YAML file (:download:`oms-daemonset.yaml <../../../../k8s/logging-and-monitoring/oms-daemonset.yaml>`): .. code-block:: yaml # oms-daemonset.yaml apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: omsagent spec: template: metadata: labels: app: omsagent spec: containers: - env: - name: WSID value: - name: KEY value: image: microsoft/oms name: omsagent ports: - containerPort: 25225 protocol: TCP securityContext: privileged: true volumeMounts: - mountPath: /var/run/docker.sock name: docker-sock volumes: - name: docker-sock hostPath: path: /var/run/docker.sock To deploy the OMS agents (one per Kubernetes node, i.e. one per computer), simply run the following command: .. code-block:: bash $ kubectl create -f oms-daemonset.yaml Search the OMS Logs ------------------- OMS should now be getting, storing and indexing all the logs from all the containers in your Kubernetes cluster. You can search the OMS logs from the Azure Portal or the OMS Portal, but at the time of writing, there was more functionality in the OMS Portal (e.g. the ability to create an Alert based on a search). There are instructions to get to the OMS Portal in the section titled :ref:`Deploy the OMS Agents` above. Once you're in the OMS Portal, click on **Log Search** and enter a query. Here are some example queries: All logging messages containing the strings "critical" or "error" (not case-sensitive): ``Type=ContainerLog (critical OR error)`` .. note:: You can filter the results even more by clicking on things in the left sidebar. For OMS Log Search syntax help, see the `Log Analytics search reference `_. All logging messages containing the string "error" but not "404": ``Type=ContainerLog error NOT(404)`` All logging messages containing the string "critical" but not "CriticalAddonsOnly": ``Type=ContainerLog critical NOT(CriticalAddonsOnly)`` All logging messages from containers running the Docker image bigchaindb/nginx_3scale:1.3, containing the string "GET" but not the strings "Go-http-client" or "runscope" (where those exclusions filter out tests by Kubernetes and Runscope): ``Type=ContainerLog Image="bigchaindb/nginx_3scale:1.3" GET NOT("Go-http-client") NOT(runscope)`` .. note:: We wrote a small Python 3 script to analyze the logs found by the above NGINX search. It's in ``k8s/logging-and-monitoring/analyze.py``. The docsting at the top of the script explains how to use it. Create an Email Alert --------------------- Once you're satisfied with an OMS Log Search query string, click the **🔔 Alert** icon in the top menu, fill in the form, and click **Save** when you're done. Some Useful Management Tasks ---------------------------- List workspaces: .. code-block:: bash $ az resource list \ --resource-group resource_group \ --resource-type Microsoft.OperationalInsights/workspaces List solutions: .. code-block:: bash $ az resource list \ --resource-group resource_group \ --resource-type Microsoft.OperationsManagement/solutions Delete the containers solution: .. code-block:: bash $ az group deployment delete --debug \ --resource-group resource_group \ --name Microsoft.ContainersOMS .. code-block:: bash $ az resource delete \ --resource-group resource_group \ --resource-type Microsoft.OperationsManagement/solutions \ --name "Containers(work_space)" Delete the workspace: .. code-block:: bash $ az group deployment delete --debug \ --resource-group resource_group \ --name Microsoft.LogAnalyticsOMS .. code-block:: bash $ az resource delete \ --resource-group resource_group \ --resource-type Microsoft.OperationalInsights/workspaces \ --name work_space .. _oms-k8s-references: References ---------- * `Monitor an Azure Container Service cluster with Microsoft Operations Management Suite (OMS) `_ * `Manage Log Analytics using Azure Resource Manager templates `_ * `azure commands for deployments `_ (``az group deployment``) * `Understand the structure and syntax of Azure Resource Manager templates `_ * `Kubernetes DaemonSet`_ .. _Azure Resource Manager templates: https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-authoring-templates .. _Kubernetes DaemonSet: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/