• 欢迎访问蜷缩的蜗牛博客 蜷缩的蜗牛
  • 微信搜索: 蜷缩的蜗牛 | 联系站长 kbsonlong@qq.com
  • 如果您觉得本站非常有看点,那么赶紧使用Ctrl+D 收藏吧

[置顶] Kubernetes集群监控

Docker 蜷缩的蜗牛 8个月前 (01-23) 51次浏览 已收录

1. 概述

1.1. cAdvisor

cAdvisor 对 Node 机器上的资源及容器进行实时监控和性能数据采集,包括 CPU 使用情况、内存使用情况、网络吞吐量及文件系统使用情况,cAdvisor 集成在 Kubelet 中,当 kubelet 启动时会自动启动 cAdvisor,即一个 cAdvisor 仅对一台 Node 机器进行监控。kubelet 的启动参数–cadvisor-port 可以定义 cAdvisor 对外提供服务的端口,默认为 4194。可以通过浏览器访问。项目主页:http://github.com/google/cadvisor
更多可参考:http://blog.csdn.net/huwh_/article/details/77918507

1.2. Heapster

是对集群中的各个 Node、Pod 的资源使用数据进行采集,通过访问每个 Node 上 Kubelet 的 API,再通过 Kubelet 调用 cAdvisor 的 API 来采集该节点上所有容器的性能数据。由 Heapster 进行数据汇聚,保存到后端存储系统中,例如 InfluxDB,Google Cloud Logging 等。项目主页为:https://github.com/kubernetes/heapster
更多可参考:http://blog.csdn.net/huwh_/article/details/77899229

1.3. InfluxDB

是分布式时序数据库(每条记录带有时间戳属性),主要用于实时数据采集、事件跟踪记录、存储时间图表、原始数据等。提供 REST API 用于数据的存储和查询。项目主页为http://InfluxDB.com
更多可参考:http://blog.csdn.net/huwh_/article/details/77899197

1.4. Grafana

通过 Dashboard 将 InfluxDB 的时序数据展现成图表形式,便于查看集群运行状态。项目主页为http://Grafana.org

1.5. 总体架构图

这里写图片描述

其中当前 Kubernetes 中,Heapster、InfluxDB、Grafana 均以 Pod 的形式启动和运行。Heapster 与 Master 需配置安全连接。

2. 部署与使用

2.1. cAdvisor

kubelet 的启动参数–cadvisor-port 可以定义 cAdvisor 对外提供服务的端口,默认为 4194。可以通过浏览器访问。也提供了 REST API 供客户端远程调用,API 返回的格式为 JSON,可以采用 URL 访问:http://:/api///

例如:http://14.152.49.100:4194/api/v1.3/machine 获取主机信息。

2.2. Service

2.2.1. heapster-service

heapster-service.yaml

<code class="language-yaml hljs avrasm"><span class="hljs-label">apiVersion:</span>v1
<span class="hljs-label">kind:</span>Service
<span class="hljs-label">metadata:</span>
  label:
    kubenetes<span class="hljs-preprocessor">.io</span>/cluster-service:<span class="hljs-string">"true"</span>
    kubernetes<span class="hljs-preprocessor">.io</span>/name:Heapster
  name:heapster
  namespace:kube-system
<span class="hljs-label">spec:</span>
  ports:
    - port:<span class="hljs-number">80</span>
      targetPort:<span class="hljs-number">8082</span>
  selector:
    k8s-app:heapster</code>

2.2.2. influxdb-service

influxdb-service.yaml

<code class="language-yaml hljs css"><span class="hljs-tag">apiVersion</span><span class="hljs-pseudo">:v1</span>
<span class="hljs-tag">kind</span><span class="hljs-pseudo">:Service</span>
<span class="hljs-tag">metadata</span>:
  <span class="hljs-tag">label</span><span class="hljs-pseudo">:null</span>
  <span class="hljs-tag">name</span><span class="hljs-pseudo">:monitoring-InfluxDB</span>
  <span class="hljs-tag">namespace</span><span class="hljs-pseudo">:kube-system</span>
<span class="hljs-tag">spec</span>:
  <span class="hljs-tag">type</span><span class="hljs-pseudo">:Nodeport</span>
  <span class="hljs-tag">ports</span>:
    <span class="hljs-tag">-</span> <span class="hljs-tag">name</span><span class="hljs-pseudo">:http</span>
      <span class="hljs-tag">port</span><span class="hljs-pseudo">:80</span>
      <span class="hljs-tag">targetPort</span><span class="hljs-pseudo">:8083</span>
    <span class="hljs-tag">-</span> <span class="hljs-tag">name</span><span class="hljs-pseudo">:api</span>
      <span class="hljs-tag">port</span><span class="hljs-pseudo">:8086</span>
      <span class="hljs-tag">targetPort</span><span class="hljs-pseudo">:8086</span>
      <span class="hljs-tag">Nodeport</span><span class="hljs-pseudo">:8086</span>
  <span class="hljs-tag">selector</span>:
    <span class="hljs-tag">name</span><span class="hljs-pseudo">:influxGrafana</span></code>

2.2.3. grafana-service

grafana-service.yaml

<code class="language-yaml hljs avrasm"><span class="hljs-label">apiVersion:</span>v1
<span class="hljs-label">kind:</span>Service
<span class="hljs-label">metadata:</span>
  label:
    kubenetes<span class="hljs-preprocessor">.io</span>/cluster-service:<span class="hljs-string">"true"</span>
    kubernetes<span class="hljs-preprocessor">.io</span>/name:monitoring-Grafana
  name:monitoring-Grafana
  namespace:kube-system
<span class="hljs-label">spec:</span>
  type:Nodeport
  ports:
      port:<span class="hljs-number">80</span>
      targetPort:<span class="hljs-number">8080</span>
      Nodeport:<span class="hljs-number">8085</span>
  selector:
    name:influxGrafana</code>

使用 type=NodePort 将 InfluxDB 和 Grafana 暴露在 Node 的端口上,以便通过浏览器进行访问。

2.2.4. 创建 service

<code class="language-shell hljs lasso">kubectl create <span class="hljs-attribute">-f</span> heapster<span class="hljs-attribute">-service</span><span class="hljs-built_in">.</span>yaml
kubectl create <span class="hljs-attribute">-f</span> InfluxDB<span class="hljs-attribute">-service</span><span class="hljs-built_in">.</span>yaml
kubectl create <span class="hljs-attribute">-f</span> Grafana<span class="hljs-attribute">-service</span><span class="hljs-built_in">.</span>yaml</code>

2.3. ReplicationController

2.3.1. influxdb-grafana-controller

influxdb-grafana-controller-v3.yaml

<code class="language-yaml hljs haml">apiVersion:v1
kind:ReplicationController
metadata:
  name:monitoring-influxdb-grafana-v3
  namespace:kube-system
  labels:
    k8s-app:influxGrafana
    version:v3
    kubernetes.io/cluster-service:"true
spec:
  replicas:1
  selector:
    k8s-app:influxGrafana
    version:v3
  template:
    metadata:
      labels:
        k8s-app:influxGrafana
        version:v3
        kubernetes.io/cluster-service:"true
    spec:
      containers:
        -<span class="ruby"> <span class="hljs-symbol">image:</span>gcr.io/google_containers/<span class="hljs-symbol">heapster_influxdb:</span>v<span class="hljs-number">0</span>.<span class="hljs-number">5</span>
</span>          name:influxdb
          resources:
            limits:
              cpu:100m
              memory:500Mi
            requests:
              cpu:100m
              memory:500Mi
          ports:
            -<span class="ruby"> <span class="hljs-symbol">containerPort:</span><span class="hljs-number">8083</span>
</span>            -<span class="ruby"> <span class="hljs-symbol">containerPort:</span><span class="hljs-number">8086</span>
</span>          volumeMounts:
            -<span class="ruby"><span class="hljs-symbol">name:</span>influxdb-persistent-storage
</span>             mountPath:/data
        -<span class="ruby"> <span class="hljs-symbol">image:</span>grc.io/google_containers/<span class="hljs-symbol">heapster_grafana:</span>v2.<span class="hljs-number">6.0</span>-<span class="hljs-number">2</span>
</span>          name:grafana
          resources:
            limits:
              cpu:100m
              memory:100Mi
            requests:
              cpu:100m
              memory:100Mi
          env:
            -<span class="ruby"> <span class="hljs-symbol">name:</span><span class="hljs-constant">INFLUXDB_SERVICE_URL</span>
</span>              value:http://monitoring-influxdb:8086
            -<span class="ruby"> <span class="hljs-symbol">name:</span><span class="hljs-constant">GF_AUTH_BASIC_ENABLED</span>
</span>              value:"false"
            -<span class="ruby"> <span class="hljs-symbol">name:</span><span class="hljs-constant">GF_AUTH_ANONYMOUS_ENABLED</span>
</span>              value:"true"
            -<span class="ruby"> <span class="hljs-symbol">name:</span><span class="hljs-constant">GF_AUTH_ANONYMOUS_ORG_ROLE</span>
</span>              value:Admin
            -<span class="ruby"> <span class="hljs-symbol">name:</span><span class="hljs-constant">GF_SERVER_ROOT_URL</span>
</span>              value:/api/v1/proxy/namespace/kube-system/services/monitoring-grafana/
          volumeMounts:
            -<span class="ruby"> <span class="hljs-symbol">name:</span>grafana-persistent-storage
</span>              mountPath:/var
      volumes:
        -<span class="ruby"> <span class="hljs-symbol">name:</span>influxdb-persistent-storage
</span>          emptyDir{}
        -<span class="ruby"> <span class="hljs-symbol">name:</span>grafana-persistent-storage
</span>          emptyDir{}</code>

2.3.2. heapster-controller

heapster-controller.yaml

<code class="language-yaml hljs haml">apiVersion:v1
kind:ReplicationController
metadata:
    labels:
        k8s-app:heapster
        name:heapster
        version:v6
    name:heapster
    namespace:kube-system
spec:
    replicas:1
    selector:
        name:heapster
        k8s-app:heapster
        version:v6
    template:
        metadata:
            labels:
                k8s-app:heapster
                version:v6
        spec:
            containers:
                -<span class="ruby"> <span class="hljs-symbol">image:</span>gcr.io/google_containers/<span class="hljs-symbol">heapster:</span>v<span class="hljs-number">0</span>.<span class="hljs-number">17.0</span>
</span>                  name:heapster
                  command:
                    -<span class="ruby"> /heapster
</span>                    -<span class="ruby"> --source=<span class="hljs-symbol">kubernetes:</span><span class="hljs-symbol">http:</span>/<span class="hljs-regexp">/192.168.1.128:8080?inClusterConfig=flase&kubeletHttps=true&useServiceAccount=true&auth=
</span></span>                    -<span class="ruby"> --sink=<span class="hljs-constant">InfluxDB</span><span class="hljs-symbol">:http</span><span class="hljs-symbol">://monitoring-InfluxDB</span><span class="hljs-symbol">:</span><span class="hljs-number">8086</span></span></code>

Heapster 设置启动参数说明:

1、–source

配置监控来源,本例中表示从 k8s-Master 获取各个 Node 的信息。在 URL 的参数部分,修改 kubeletHttps、inClusterConfig、useServiceAccount 的值。

2、–sink

配置后端的存储系统,本例中使用 InfluxDB。URL 中主机名的地址是 InfluxDB 的 Service 名字,需要 DNS 服务正常工作,如果没有配置 DNS 服务可使用 Service 的 ClusterIP 地址。

2.3.3. 创建 ReplicationController

<code class="language-shell hljs lasso">kubelet create <span class="hljs-attribute">-f</span> InfluxDB<span class="hljs-attribute">-Grafana</span><span class="hljs-attribute">-controller</span><span class="hljs-built_in">.</span>yaml
kubelet create <span class="hljs-attribute">-f</span> heapster<span class="hljs-attribute">-controller</span><span class="hljs-built_in">.</span>yaml</code>

3. 查看界面及数据

3.1. InfluxDB

访问任意一台 Node 机器的 30083 端口。

3.2. Grafana

访问任意一台 Node 机器的 30080 端口。

4. 容器化部署

4.1. 拉取镜像

<code class="language-shell hljs avrasm">docker pull registry<span class="hljs-preprocessor">.wae</span><span class="hljs-preprocessor">.haplat</span><span class="hljs-preprocessor">.net</span>/test/influxdb:<span class="hljs-number">1.0</span><span class="hljs-number">.0</span>
docker pull registry<span class="hljs-preprocessor">.wae</span><span class="hljs-preprocessor">.haplat</span><span class="hljs-preprocessor">.net</span>/test/cadvisor:latest
docker pull registry<span class="hljs-preprocessor">.wae</span><span class="hljs-preprocessor">.haplat</span><span class="hljs-preprocessor">.net</span>/test/grafana:latest
docker pull registry<span class="hljs-preprocessor">.wae</span><span class="hljs-preprocessor">.haplat</span><span class="hljs-preprocessor">.net</span>/test/heapster:latest</code>

4.2. 运行容器

4.2.1. influxdb

<code class="language-shell hljs lasso"><span class="hljs-variable">#influxdb</span>
docker run <span class="hljs-attribute">-d</span> <span class="hljs-attribute">-p</span> <span class="hljs-number">8083</span>:<span class="hljs-number">8083</span> <span class="hljs-attribute">-p</span> <span class="hljs-number">8086</span>:<span class="hljs-number">8086</span> <span class="hljs-subst">--</span>expose <span class="hljs-number">8090</span> <span class="hljs-subst">--</span>expose <span class="hljs-number">8099</span> <span class="hljs-subst">--</span>volume<span class="hljs-subst">=</span>/opt/<span class="hljs-built_in">data</span>/influxdb:/<span class="hljs-built_in">data</span> <span class="hljs-subst">--</span>name influxsrv registry<span class="hljs-built_in">.</span>wae<span class="hljs-built_in">.</span>haplat<span class="hljs-built_in">.</span>net/test/influxdb:<span class="hljs-number">1.0</span><span class="hljs-number">.0</span></code>

4.2.2. cadvisor

<code class="language-shell hljs ruby"><span class="hljs-comment">#cadvisor</span>
docker run --volume=<span class="hljs-regexp">/:/rootfs</span><span class="hljs-symbol">:ro</span> --volume=<span class="hljs-regexp">/var/run</span><span class="hljs-symbol">:/var/run</span><span class="hljs-symbol">:rw</span> --volume=<span class="hljs-regexp">/sys:/sys</span><span class="hljs-symbol">:ro</span> --volume=<span class="hljs-regexp">/var/lib</span><span class="hljs-regexp">/docker/</span><span class="hljs-symbol">:/var/lib/docker</span><span class="hljs-symbol">:ro</span> --publish=<span class="hljs-number">8080</span><span class="hljs-symbol">:</span><span class="hljs-number">8080</span> --detach=<span class="hljs-keyword">true</span> --link <span class="hljs-symbol">influxsrv:</span>influxsrv --name=cadvisor registry.wae.haplat.net/test/<span class="hljs-symbol">cadvisor:</span>latest -storage_driver=influxdb -storage_driver_db=cadvisor -storage_driver_host=<span class="hljs-symbol">influxsrv:</span><span class="hljs-number">8086</span></code>

4.2.3. grafana

<code class="language-shell hljs lasso"><span class="hljs-variable">#grafana</span>
docker run <span class="hljs-attribute">-d</span> <span class="hljs-attribute">-p</span> <span class="hljs-number">3000</span>:<span class="hljs-number">3000</span> <span class="hljs-attribute">-e</span> INFLUXDB_HOST<span class="hljs-subst">=</span>influxsrv <span class="hljs-attribute">-e</span> INFLUXDB_PORT<span class="hljs-subst">=</span><span class="hljs-number">8086</span> <span class="hljs-attribute">-e</span> INFLUXDB_NAME<span class="hljs-subst">=</span>cadvisor <span class="hljs-attribute">-e</span> INFLUXDB_USER<span class="hljs-subst">=</span>root <span class="hljs-attribute">-e</span> INFLUXDB_PASS<span class="hljs-subst">=</span>root <span class="hljs-subst">--</span><span class="hljs-keyword">link</span> influxsrv:influxsrv <span class="hljs-subst">--</span>name grafana registry<span class="hljs-built_in">.</span>wae<span class="hljs-built_in">.</span>haplat<span class="hljs-built_in">.</span>net/test/grafana:latest</code>

4.2.4. heapster

<code class="language-shell hljs bash"><span class="hljs-comment">#heapster</span>
docker run <span class="hljs-operator">-d</span> -p <span class="hljs-number">8082</span>:<span class="hljs-number">8082</span> --net=host heapster:canary --source=kubernetes:http://<k8s-server-ip>:<span class="hljs-number">8080</span>?<span class="hljs-keyword">in</span>ClusterConfig=<span class="hljs-literal">false</span>\&useServiceAccount=<span class="hljs-literal">false</span> --sink=influxdb:http://<influxdb-ip>:<span class="hljs-number">8086</span></code>

4.3. 访问

在浏览器输入 IP:PORT

本文转载自
[置顶]
Kubernetes 集群监控


蜷缩的蜗牛 , 版权所有丨如未注明 , 均为原创丨 转载请注明[置顶] Kubernetes 集群监控
喜欢 (0)
[]
分享 (0)