背景
环境信息如下:
- kube-apiserver、kube-controller-manager、kube-scheduler 通过 systemd 手工部署
- Promethus 运行在 k8s 集群中,容器是自定义的并未部署 Operator
- 对接以下服务的 metrics
- kube-apiserver
- kube-controller-manager
- kube-scheduler
- kubelet/metric-server(metric-server 提供 kubelet 和自定义指标的聚合,本身不产生指标)
- kube-state-metrics
Kubernetes 版本 1.22.3 ,Promethus 的 ServiceAccount 权限如下:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
|
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: promethus
rules:
- apiGroups: [""]
resources:
- nodes
- nodes/stats
- nodes/metrics
- nodes/proxy
- services
- endpoints
- pods
- ingress
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: promethus
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: promethus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: promethus
subjects:
- kind: ServiceAccount
name: promethus
namespace: default
|
通过 CURL 验证接口
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
# 获取 token
SECRET_NAME=`kubectl get secret |grep promethus-token |awk '{print $1}'`
TOKEN=`kubectl get secret $SECRET_NAME -o jsonpath="{['data']['token']}" |base64 -d`
# kube-apiserver
# 这里通过 -ik 参数跳过了 client 对 kube-apiserver 私签证书的校验,因此不需要通过 --cacert 来制定 ca 文件
curl -k -H "Authorization: Bearer $TOKEN" https://127.0.0.1:6443/metrics
# 通过 kube-apiserver 我们可以代理所有 Node 上的 metrics 信息,如下
curl -k -H "Authorization: Bearer $TOKEN" https://127.0.0.1:6443/api/v1/nodes/<NODE_HOST_NAME>/proxy/metrics
# kubelet
# 我们也可以不通过代理,直接从 kubelet 端口抓去数据
curl -k -H "Authorization: Bearer $TOKEN" https://127.0.0.1:10250/metrics
curl -k -H "Authorization: Bearer $TOKEN" https://127.0.0.1:10250/metrics/cadvisor
curl -k -H "Authorization: Bearer $TOKEN" https://127.0.0.1:10250/metrics/resource
curl -k -H "Authorization: Bearer $TOKEN" https://127.0.0.1:10250/metrics/probes
# kube-controller-manager
curl -k -H "Authorization: Bearer $TOKEN" https://127.0.0.1:10257/metrics
# kube-scheduler
curl -k -H "Authorization: Bearer $TOKEN" https://127.0.0.1:10259/metrics
|
kube-controller-manager 和 kube-scheduler 服务需要通过 –bind-address=0.0.0.0 配置对外暴露服务端口。
当基于 RABC 权限校验获取数据时,需要额外配置 –authentication-kubeconfig 和 –authorization-kubeconfig 参数。
用户也可以通过 –authorization-always-allow-paths 将 /metrics 的认证关闭。
1
|
--authorization-always-allow-paths=[/healthz,/readyz,/livez,/metrics]
|
Promethus 配置
以下 job 筛选集群中所有的 master 节点内网 IP 地址,并从采集 kube-apiserver 信息,其他 kubernetes 组件配置类似
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
|
- job_name: kube-apiserver
honor_timestamps: true
scrape_interval: 1m
scrape_timeout: 10s
metrics_path: /metrics
scheme: https
kubernetes_sd_configs:
- role: node
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
insecure_skip_verify: true
relabel_configs:
- separator: ;
regex: __meta_kubernetes_node_label_(.+)
replacement: $1
action: labelmap
- source_labels: [__meta_kubernetes_node_labelpresent_node_role_kubernetes_io_master]
separator: ;
regex: "true"
replacement: $1
action: keep
- source_labels: [__meta_kubernetes_node_address_InternalIP]
separator: ;
regex: (.+)
target_label: __address__
replacement: $1:6443
action: replace
|
参考