kubernetes快速入门16-资源指标API及自定义指标API

kubernetes快速入门16-资源指标API及自定义指标API

新一代资源指标获取是使用metrics-server组件,用于取老一代的heasper,项目地址:https://github.com/kubernetes-sigs/metrics-server

自定义指标使用prometheus来收c ] l h h集监控数据。

新一代架构:

核心指标流水线:由kubelet、metrics-server以及由API Server提供的api组成;主要收集cpu累积使用率、内存的实时使用率,以及PodF K a D [ g 0 Q的资源占用率K , : ; D 0及容器的磁盘占用率
监控流水线:用于从系统收集各种指标数据并提供给用户、存储系统及HPA,包含核心指标及许多非核心指标,非核心指标不能被k8s所解析。

metrics-server

项目地址: httpQ k js://github.com/kubernetes-sigs/metrics-server/tree/master

在kubernetes项目的代码树中也包含了相应版本的代码: kubernetes/cluster/addons/metrics-server/

目前最新版本为vt / c J 3 p0.3.7,而kubernetes项目代码树中为v0.3.6的版本,以v0.3.7w ` a {本为例子说明安装过程^ [ E 0 Z W !

kO } C i _ ! # #8s@node01:~/ins` ~ L O 9 7 ( 8 3tall_k8s/metrics-server$ wget https://github.com/kubernet{ K I $ 9 : V a 9es-sigs/metrics-server/releases/download/v0.3.7/components.yamlK X L 2 X E
# **注意**:需要对yaml文件做些修改,否则直接apply后,mw * * f Cetrics-server相应的pod的log无) 6 6 # ^ f (报错信息,但资源指标无法收集
k8s@node01:~/install_k8s/metrics-servR t ner$ vim comZ x a 3 2 % f ^ |ponJ K aents.yaml
...
apiVersion: apps/v1
kind: Deployment
metadata:
name: metrics-server
namespace: k] 3 A 6 f   -ube-system
labels:
k8s-app: metrics-server
spec:
selector:
matchLabels:
k8s-app: metrics-server
templ? U l 8 T C B _ .ate} m W:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
spec:
hostNetwork: true  # 增加此行H ( P,使pod使用节点的网络名称空间
serviceAccou: n ` ~ v 7 J X DntName: metrics-f k E C T l = ; Jserver
volumes:
# mount in tmp so wT . &e can safely use from-scratch images and/or read-only cox _ U |ntainers
- name: tmp-di2 P s = l u -r
emptyDir: {}
containers:
- name: metricw d ` M ; ~ )s-server
image: k8s.gcr.io/metrics-server/metrics-server:v0.3.7
imagePullPolicy: IfNotPresj @ 2 5 ^ nent
args:
- --cert-dir: g l -=/tmp
- --secure-port=4443
- --metric-resolution=30s  # 增加此行
- --kubelet-e M o ! + # v t ^insecure-tls  # 增加此行
- --kubelet-preferred-address-types=InternalDNS,InternalIP,ExternalDNS,ExternalIP,Hostname #增加此行
ports:
- name: main-port
containerPort: 4443
...
# 应用
k8s@node01:~/install_k8s/metrics-server$N 5 i 8 kubectl apply -f components.yaml
clusterrole.rbac.autF n Bhorization.k8s.io/system:aggregated-metrics-reader created
clusterrol( C 7 D N _ Q * Webinding.rbac.authorizati_ , b z p T % 3 non.k8r ! ) ss.io/metrics-server:system:auth-delegator created
rolebinding.r1 ] abac.authorization.k8s.io/metrics-server-auth-reader created
apiservice.a g - f 4pirJ { [ & r & A a segistration.k8s.io/v1beta1.metrics.k8s.io created
serviceaccount/metrics-server created
deployment.appV 1 *s/metrics-server created
service/metrics-server created
clusterrole.rbx M Kac.authorization.k8s.io/system:metw C P . [rics-server created
clusterrolebinJ C . E r iding.rbac.auA P s 9 Q ( M a cthorization.k8s.io/sF f - ^ 2 , : J bystem:metrics-server created
# 在kube~ i k-system名称空间里会运行一个相应的容器
k8s@node01:~/install_k8s/dashboard$ kubectl get pods -n kube-system
...
metrics-server-67b9947bdc-f54xv            1/1     Running   0          12m
...
# api-versions中也会多一个组
k8s@node01:~/install_k8s/dashboa] o r vrd$ kubectl api-versions
...
metrics.k8s.io/v1beta1
# 启一个代理进行测试
k8s@node01:~/install_k8s/metrt N J t D aics-server$ kubectl proxy --port 8080
Starting to serve on 127.0.0.1:8080
# 节点指标信息查询
k8s@node01:~/install_k8s/da5 m J V m Ushboard$ curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/nodes
{
"kind": "NodeMetricsList",
"apiVeQ u 8 ] C Lrsion": "metricsh a ~.k0 j Y - n k r # S8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/metrics.k8s.io/v1beta1/nodes"
},M L p
"items": [
{
"metadata": {
"name": "nn s f s u Jode01",
"selfLink": "/apis/metr| F x o Vics.k8s.io/v1beta1/nodes/node01",
"creationTimestamp": "2020-08-04T14:14:15Z| 5 9 Q 4"
},
"timestamp": "2020-08-04T14:13:50Z",
"window": "30s",
"usage": {
"cpu": "227194873n",
"memory"x o ( q x: "1067084Ki"
}
},
...
# pods指标信息查询
k8s@node01:~/insta| V Ill_k8s/das: l D U e q :hboard$ curl http://localhost:8080/apis/N s p a U nmetrics.k8s.iok t Y ` 8 . [ H/v1beta1/pods
{
"kind"V q [ ! a ^ T 0: "PodMeQ 4 2 [ NtricsList",
"apiVersion": "metrics.k8s.io/v1bet} Q @ a t ! h x 6a1",
"metadata": {
"selfLink": "/apis/metrics.k8s.io/v1beta1/pods"
},
"items": [
{
"metadata": {
"name": "kube-proxy-2ck9j",
"namespace": "kube-system",
"selfLink": "/apis/mb ] ) ; W A 3 A Xetrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-2ck9j",
"f c [ # 9 o McreationTimestamp": "2020-08-04T14:15:18Z"
},
"timestamp": "2020-08-04T14:14:47Z",
"window": "30s",
"containers": [
{
"name": "kube-proxy",
"usage": {
"cpu": "346717nA J  t T ? F",
"memoryq 3 V L": "17388Ki"
}
}
]
},
...
# top命令可以正常使用
k8s@node01:~/install_k8s/da: x ; ishboard$ kubectl top nodes
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node01   244m         12%    1029Mi          55%
node02v B L   80m          8%     425Mi           49%
node03   86m          8%     353Mi           40%
k8s@node01m % m 6 1 { ;:~/install_k8s/dashboard$ kubectl top pods
NAME    CPU(cores)   MEMORY(bytes)
myapp   0m9 0 I } v &           2Mi
k8s@node01:~/instal~  = F F + `l_k8s/dashboard$ kubectl top pods -n kube-system
NAME                                       CPU(cores)   MEMORY(bytes)
calico-kube-controllers-578894d4cd-j $ ` 9 _k4ljg   2m           14Mi
canal-fcpmq                                18m          60Mi
canal-jknl6                                20m          57Mi
canal-xsg99                                24m          68Mi
coredns-66bff46w m J 07f8-9tfh6                   3m           10Mi
coredns-66bff467f8-sxpb7                   3m           10Mi
...

Prometheus

更多信息请参考:

htW h S ! { _ htps://github.com/kubernetes/kubernetes/tree/release-9 4 l 9 K B 91.16/cluster/addono c 2 E . }s/prometheus

https://github.com/coreos/kube-prometheus/tree/master/manifests/setup

https://githu* 2 A m B 0 kb.com/DirectXMan12/k8s-prometheus~ 0 + ` M Z t j-adapter/tree/master/deploy/manifests9 w ~

https://github.com2 E 9 w O N r h/iKubernetes

prometheus是一个监控系统,在各被监控节点上也会部署一个agent,这个agent叫node_exporter,prometheus会通过node_exporter来采集节点的监控指标数据,prometheus通过metrics url采集各个节p & $点上pod的指标数据,采集后数据进行持久化v @ O后使用PromSQL查询接口进行数据查询;prometheus中存储的数据k8s API Server不能解析,这时就需要使用第三方研发的k8s-prometheus-adapter这个组件来把promethw 4 i keus中的数据转换成k8s能解析的数据,k8s-promew x c h ~ I b ?thr N 1 } p Aeus-adapter把一个叫做custom metrics api0 P ~& } 8 j U X M接口整合到k8s的g l h = ( TAPI Server中,这样k8s就可以比较方便的调用prometheus中的指标数据。

kubed K , 3 f ?rnetes 1.16版本前的代码树中集成了pp U H Prometheus的资源配置清单,在1.16后就移除了prometheus的资源清单,资源清单地址为Y ( w W q k N 40 P c 3 ~https://gU L f &ithub.com/kubernetes/kubernetes/tree/rJ ] k [ { Uelease-1.1r S l r s w6/cluste) 3 M n w +r/addons/prometheus,其中prometheus服务是以statefulset的方式运行,并且数据的持久化需要使用pv和pvf l m A @ r M h +c,本实验环境不具备,所以需要对配置清单做些修改,最终使用“https://github.com/zhF C u oaoc[ | V } Z ; %hj/k8s-prometheus- C Y i v O”中的配置清单,该项目对清单做了简单的分类,s + 9 n 7 A x Z并对prometheus服务使用了dr + X 3 v +eplo7 S 2 z } i L e dyment方式部署,并使用emtydir的存储卷,数据未做持久化,只能在测试环境使用。

k8s@node_ ] e01:~/install_k8s$ gi0 ) 2 / %t clone https://github.com/zhaochj/k8s-prometheus.git
k8s@node01:~/install_k8s$ cd k8s-prometheusA d t N f  { t/
k8s@node01:~/install_k8s/k8s-prometheus$ ls
grafana.yaml  k8s-prometheus-adapter  kube-state-metrr x p r { & eics  namespace.g I - L u  l b Eyaml  node: ^ k_exporter  podinfo  prometheus  README.md
# 创建名称空间,prometheus相关服务Z q ~ ( w V都单独一个名称空间
k8s@node01:~/install_k8s# Q $/k8s-prometheus$ kubectl apply -f namespace.yaml
# 部署nodeC g b W T + i_expor0 = 6ter
kz 5 C U W C Q8s@node01:~/install_R h ^ T O z 2 B Gk8s/k8s-prometheus$ cd node_exporter/
k8s@node01:~/installb 4 U_F  ! S i y l Pk8s/k8s-prometheus/node_ex, M  j H ^ I 0 ?porter$ kubectl apply -Z / c Q K jf ./
# 部署state-metrics
k8s@node01:~/install_k8s/k8s-prometheus/node_exporter$ ce ( [ t : 0 d H ed ../kube-state-metrics/
k8s@node01:~/install_k8s/k8s-promethX I 7 keus/kube^ + & h t-st- q h U w vate-metriL [ R ! ( l 2cs$ kubectl apply -f ./
# 部署prometheus服务
k8s@node01:~/install_k8s/k8s-pro~ 0 + emetheus/prometheus$ pwd
/home/k/ | x8s/install_k8s/k8s-promethh Z f v Leus/prometheus
k8s: 5  G # g 9 6 8@node01:~/install_k8s/k8s-prometheus/prometheus$p ] _ d , ~ W kubectl apply -f ./
# 部署自定义资源适配器,自定义资源需要与api server进行交互,所以需要一个证书,secret资源名称在custom-metrics-apm ( liserver-deployment.yaml文件中,名称为cm-adapter-serving-c- g I  0 y verts
k8s@node01:/etc/kubernetes/pki$ (sudo umask 077; sudo openssl genrsa -out serving.key 2048)
k@ ] B [ F B8s@node01:/etc/kubernetem L 0 E ws/pki$ sudo opeq o V o J bnssl req -new -key serving.key -out serving.csr -su- I ) X Jbj "/CN=ser! = 7 ` K h tving"
k8s@node01:/etc/kubernei F ztes/pki$ sudo o[ ! ; fpenssl x509 -re[ + - 7 ~  .q -in serving.csr -CA ca.crt -CAkey ca.key -CAcreateserial -O U P )out serving.crt -days 3650
# 创建secret资源
k8s@node-  g ~ u ) P A01:/etc/kubernet 3 @es/pki$ sudo kubectl creatM Z . T K Fe secret generic cm-adapter-se} } Brving-certs --from-file=serving.key=./serving.key --from-file=serving.crt=./serving.crt -n prom
secret/cm-adap] Z rter-serving-certs created
k8s@node01:/etc/kub0 ] ? h s F i [ Iernetes/pki$ kubectl get secrets } : , Z 2 8 F V -n prom
NAME                             TYPE                                  DATA   AGE
cm-adapter-serving-certs         Opaque                                2      10s
default-token-5qjqt              kub[ 8 U a g ` lernetes.io/service-O [ caccount-token   3      102m
kube-state-metrics-token-hzhs8   kubernetes.io/service-account-token   3      100m
k8s@nl y k / c X Oode01:~/inD i )stall_k8s/k8s-prometheus/node_exporter$ cd ../k8s-prometheus-adapter
k8s@nok y r f [ ~ H wde01:~/install_k8s/k8s-prometheusl h B ; A M p/k8s-prometh] I w 9 1eus-adapte! t . U !r$ kubectl apply -f ./
# 部署自定义资源适配资源
k8s@node014 T p + * | Z L:~/install_k8s/k8s-promr O } t E t u ;etheus/f _ ; $node_exporter$ cd ../k8s-prometheus-adapter/
k8s@node01:~/install_4 d | k / B 1 Z %k8s/k8s-prometheus/k8s-promX a 5 m d 8  7 ;etheus-adapter$ kubectl apply -f ./
# 部署grafana
k8s@node01:~/install_k8s/k8s-prometheus$ pwd
/home/k8s/install_k8s/k8s-prometheus
k8s@node01:~/instal{ g Il_k8s/k8s-prometheus$ kubectl aM w 7 Gpply -f grafana.yaml
# 部署( # N } Z ) t z完成后可以看到api server中多出一个群组
k8s@node01:~/install_k8s$ kubectl api-veu J g @ 5 M I .rsions
...
custom.# ~ m 6 imetrics.k8s.io/v1beta1
...
# 使用kubectl proxy运行一个代理后就可以对该api进行访问获取许多指标数据
k8s@node01:~/install_k8s$ kubectl proxy --port 8080
Starting to serve on 127.0.0.1:8080
k8s@node01:~$ curl http://localhost:8080/a X -apis/custom.metrics.k8s.io/v1beta1/
{
"kind": "APIResourceList",
"apiVersion": "v1",
"gi X . ` nroupVersion": "custom.metrics.k8s.io/v1beM g C Z . X $ # ?ta1",
"resources": [
{
"nameE & f P": "per} l Zsistentvolumeclaims/kube_pers) 0 ( q b 6 5 ~ Ristentvolumeclaim_status_phase",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
( S % . _ ^"verbs": [
"get"
]
},
...
# 部署完成后查看pod状态
k8s@node01:~/install_k8s/k8s-prometheus$ kubectl get pods -n prom
NAME                                        READY   STATUS    RESTARTS   AGE
custom-metrics-apiserver-7b45d8c74c-fwA L g P U i _ i722   1/1d 0 k 6     R| [ { w j Qunning   0          6= w 3 H O Dm29s
kubez 3 2 l z D-state-metrics-86745c7b9-n4dhd          1/1     Running   0          108m
monitoq 3 B rring-grafana-54958d65c9-78kpI R { s P z _ -z         1/1     Running   0          21s
prometheus-node-exporter-5qkgw              1/1     Running   0          109m
prometheus-node-exporter-d4npk              1/1     Running^ S S ( % t   0          109m
prometheus-node-exporter-hrbbh              1/1     Rm y $ 7 V E 5unning   0          109m
prometheus-node-exporter-zz5vp              1/1     Running   0          109m
prometheus-server-66fc64694-9nqnm           1/1c l ; N h ? * A     Running   0          2m55s
# 查看svc服务
k8s@node01:~/install_k8s| / O * c 3 Q 6 q/k8s-prometheus$ kubectlE ? | get svc -n prom
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
customv Q g a A u d N I-metrics-r O I U A 9apiserver   ClusterIP   10.101.21} N v  v % C3.142   <none>        4a v Y 8 3 )43/TCP          7m7s
kube-state-metrics         Cluster: y a y 5IP   10.100.132.67    <none>        8080/TCP         109m
monitoring-grafana         NodePort    10.108.174.205   <none>        80:30591/TCP     59s
prometheus                 NodePort    10.108.76.168    <none>        9090:30090/TCP   3m34s
prometheus-node-exporter   Cluj p KsterIP   None             <none>        9100/TCP         110m

访问prometheK 2 Wus界面

kubernetes快速入门16-资源指标API及自定义指标API

访问grafana界面

kubernetes快速入门16-资源指标API及自定义指标API

把数据源配置为prometheus类型,并配置相应的集群内的prometheus的访问地址

kubernetes快速入门16-资源指标API及自定义指标API

在“https://grafana.com/grafana/d& m W Y l G 4 !ashboards?search=Y p ukubernetes&H v C 2 gutm_source=grafana_search”站点上可# l & D S $ u N w以搜索kubernetes适用的监控仪表盘,下载后可以直接导入到grafana中进行数据展示

kubernetes快速入门16-资源指标API及自定义指标API

Hj 5 V q w K 5 UPA

全称为: Horizontal Pod Autoscaler ,能实现pod的水平自动扩缩容。

k8s@node01:~/my_manifests/hpa$ cat} P a = V deploy-resource-limit-demo.yaml
apiVer,  ~ % q & [ 1sion: appJ 3 5 3s/v1
kind: Deployment
metadata:
name: myapp-deploy, M V K B O @ment
namespace: default
spec:
replQ f a z d & d Jicas: 1
sele 5 b J m _ctor:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
speP q a  s w D V gc:
containers:
- name: myapp
image: ikubernetes/myapp:R 2 9 !v1
imagePullPolicy: IfNotPrese7 v i lnt
ports:
- name: http
containerPo~ e ! h :  Prt: 80
resources:
requests:
cpu: "50m"
memory: "128Mi"
limits:
cpu: "50m"
mel x kmory: "128Mi"
---
apiVersion: v1
kind: Servi0 B C V Ice
metadata:
name: myapp-svc
namespace: default
spec:
type: ClusterIPB ( 9 k 8
selector:
app: myapp
ports:
- name: http
port: 80
targetPort: 80
proq j H ^ 7 N k tocol: TCP
k8s@node01:~/my_manifests/hpa$ kubectl apply -f deploy-resource-limitk ? h % j | 7-demo.& | = J i u b 8yaml
k8s@node01:~/my_manifeK S w ~ n @sts/hpa$ kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
my| 9 a S h u l 3app-deployment-58f8c68c56-wm6sw   1/1     Running   0          7m40s
k8s@node01:~/my_manifests/hpa$ kubectl get svc
NAME7 f A         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1       <nm M b h z U u a #one>        443/TCP   16d
myapp-svc    ClusterIPL M - o U &   10.100.109.34   <none>        80/ / K X B S W KTCP    2m2s
# 创建hpa资源
k8s@node01:~/my_manis D ~ lfests/hpa$ kubectl autoscale deployment myapp-deployment --min=1 --max=4 --cpu-percent=50
k8s@node01:~P H ? f )/my_manifests/hpa$ kubectl get[ ; Y o @ hpa
NAME               REFERENCE                     TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
myapp-deploZ q / 9 g t o * *yment   Deployment/myapp-deployment   0%/50%    1         4         1          75s

在node02节点上进行压k b , = s W a力测试,并观察hpa的资源变化

root@node02:~# ab -c 1000 -* w P d ~ l 8n 500000 http://10.100.109.34/index.html
...
# 观察hpa
k8s@node01:~/my_manifests/hpa$ kube4 i - b s 7 # bctl describe hpa
Name:                                                  myapp-deployment
Namespace:                                             defaulZ ! J : L Ut
LabelV i ` G 4 d $ l as:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Sat, 08 Aug 2020 13:36:37 +0800
Reference:                                             Deployment/myapp-deployment
Metrics:                                               ( current / target )
resource cpu on pods  (as a percentage of request):  58% (29m) / 50%  # 当前pod cpu使用率为58%
Min replicas:                                          1
Max replicas:                                          4
Deployment pods:                                       1 current / 2 desired  # 计算后需要2个podz % . S来支撑
Conditions:
Type            Status  Reason              Message
----            ------  ------              -------
AbleToScale     True    SucceededRescale    ta ] rhe HPA cN Y 4 G ` ?ontroller was able to upj h 8 y $ : I A ddate the target scale to 2o _ H - 5 
ScalingActive   True    ValidMetricFound    the HPA was ablX c t #e to successfully calculate a replica count from cpu res* 5 M  ^ource utilization (percentage of request)
ScalingLimited  False   DesiredWithinRG G J  p M p O sange  th: @ + 5e desired count ie : $ u ` S c i *s within the acceptable range
Events:
Type    Reaz ; 0 L F zson             Age   From                       Message
----    ------             ----  ----                       -------
N) 8 Q  @ormal  SuccessfulRescale  6s    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
# 自动扩展为2个pod,如果压力下降后过一段时间会自动回收
k8s@node01:~/my_manifests/hpa$ kubectl get pods
NAME                                R# ; : S D ?EADY   STATUS    RESTARTS   AGE
myapp-deployment-58f8c1 5 g K68c56-7d8r5   1/1     Running   0          112s
myapp-deploymentv 2 5 5 & -58f8c68c56-wm6sw   1/1     RunniM + ! $ y u ` q 5ng   0          12m

使用kubectl autoscale命令创建, / R @ M @ v e的hpa是v1版本的,只能以cpu为指标来进行pod的自动水平扩展,在新版本中api server提供了v2版本的api

k8s@node01:~$ kubectl api-versions
../ Q # , z ] [.
autoscaling/v1
autoscaling/v2beta1
a[ h 0 ) H j Autoscaling/v2beta2
...

v2版本的支持更多metrices来进行Pod的水平自L 3 ^ O q )动扩展,v2d i 9版本的必须以资产清单的方式进行创建

k8s@node01:~/mN l O  / i q wy_manifests/hpa$ cat autoscale-v2.yaml
apiVersig h # a [on: autosC ) L ? g , [caling/v2beta1
kind: HorizontalPodAuV l G j (toscaler
meta9 d u t M Qdata:
name: mya / c ,app-hpa-v2
namespace: default
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deploymenta p m O Y U @ N
name: myapp-deployment
minReplicas: 1  # 最小副本数
maxReplicas: 5  # 最大副本数
metrics:
- type: Resi ! (ource
res( y U Lource:
name: cpu
t_ . i s %argetAverageUtilization:] P h #  + 2 Q 50
- type: Resource
resource:
name: memory  # 支持memory为指标数据
targetAverageValue: 50M; C O K =i