背景

环境信息如下：

  • kube-apiserver、kube-controller-manager、kube-scheduler 通过 systemd 手工部署
  • Promethus 运行在 k8s 集群中，容器是自定义的并未部署 Operator
  • 对接以下服务的 metrics
    • kube-apiserver
    • kube-controller-manager
    • kube-scheduler
    • kubelet/metric-server（metric-server 提供 kubelet 和自定义指标的聚合，本身不产生指标）
    • kube-state-metrics

Kubernetes 版本 1.22.3 ，Promethus 的 ServiceAccount 权限如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: promethus
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - nodes/stats
      - nodes/metrics
      - nodes/proxy
      - services
      - endpoints
      - pods
      - ingress
    verbs: ["get", "list", "watch"]
  - nonResourceURLs: ["/metrics"]
    verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: promethus
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: promethus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: promethus
subjects:
  - kind: ServiceAccount
    name: promethus
    namespace: default

通过 CURL 验证接口

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# 获取 token
SECRET_NAME=`kubectl get secret |grep promethus-token |awk '{print $1}'`
TOKEN=`kubectl get secret $SECRET_NAME -o jsonpath="{['data']['token']}" |base64 -d`

# kube-apiserver
# 这里通过 -ik 参数跳过了 client 对 kube-apiserver 私签证书的校验，因此不需要通过 --cacert 来制定 ca 文件
curl -k -H "Authorization: Bearer $TOKEN" https://127.0.0.1:6443/metrics

# 通过 kube-apiserver 我们可以代理所有 Node 上的 metrics 信息，如下
curl -k -H "Authorization: Bearer $TOKEN" https://127.0.0.1:6443/api/v1/nodes/<NODE_HOST_NAME>/proxy/metrics

# kubelet
# 我们也可以不通过代理，直接从 kubelet 端口抓去数据
curl -k -H "Authorization: Bearer $TOKEN" https://127.0.0.1:10250/metrics
curl -k -H "Authorization: Bearer $TOKEN" https://127.0.0.1:10250/metrics/cadvisor
curl -k -H "Authorization: Bearer $TOKEN" https://127.0.0.1:10250/metrics/resource
curl -k -H "Authorization: Bearer $TOKEN" https://127.0.0.1:10250/metrics/probes 

# kube-controller-manager
curl -k -H "Authorization: Bearer $TOKEN" https://127.0.0.1:10257/metrics

# kube-scheduler
curl -k -H "Authorization: Bearer $TOKEN" https://127.0.0.1:10259/metrics

kube-controller-manager 和 kube-scheduler 服务需要通过 –bind-address=0.0.0.0 配置对外暴露服务端口。

当基于 RABC 权限校验获取数据时，需要额外配置 –authentication-kubeconfig 和 –authorization-kubeconfig 参数。

用户也可以通过 –authorization-always-allow-paths 将 /metrics 的认证关闭。

1

--authorization-always-allow-paths=[/healthz,/readyz,/livez,/metrics]

Promethus 配置

以下 job 筛选集群中所有的 master 节点内网 IP 地址，并从采集 kube-apiserver 信息，其他 kubernetes 组件配置类似

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

- job_name: kube-apiserver
  honor_timestamps: true
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  kubernetes_sd_configs:
  - role: node
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - separator: ;
    regex: __meta_kubernetes_node_label_(.+)
    replacement: $1
    action: labelmap
  - source_labels: [__meta_kubernetes_node_labelpresent_node_role_kubernetes_io_master]
    separator: ;
    regex: "true"
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_node_address_InternalIP]
    separator: ;
    regex: (.+)
    target_label: __address__
    replacement: $1:6443
    action: replace

参考