K8S监控（kube-state-metric/cadvisor） - 监控相关

##Grafana监控图
	https://grafana.com/grafana/dashboards/13105
----
##监控k8s
###安装kube-state-metrics（k8s监控）
[kube-state-metrics安装包下载](http://baofile.frp.bao-server.club:22080/share/CIvF6FX4"kube-state-metrics安装包下载")

![](/media//202106/2021-06-08104335516072.png)

cd Prometheus-master/kubernetes
	
	### kube-state-metrics部署说明：
	# kube-state-metrics部署在ops-monit命名空间
	kubectl create namespace ops-monit
	cd kube-state-metrics
	kubectl apply -f .

###修改NodePort模式
	vim service.yaml
	#内容：
    apiVersion: v1
    kind: Service
    metadata:
    #  annotations:
    #    prometheus.io/scrape: 'true'
      labels:
        app.kubernetes.io/name: kube-state-metrics
        app.kubernetes.io/version: v1.9.7
      name: kube-state-metrics
      namespace: ops-monit
    spec:
      type: NodePort
      ports:
      - name: http-metrics
        protocol: TCP
        port: 8080
        targetPort: http-metrics
        nodePort: 32080
      - name: telemetry
        protocol: TCP
        port: 8081
        targetPort: telemetry
        nodePort: 32081
      selector:
        app.kubernetes.io/name: kube-state-metrics

###修改prometheus配置
	vim prometheus.yml
	#内容：
	  - job_name: 'k8s-status'
	    # metrics_path defaults to '/metrics'
	    #     # scheme defaults to 'http'.
	    static_configs:
	    - targets: ['172.16.16.169:32080']
	    - targets: ['172.16.16.169:32081']

###查看
	http://192.168.110.151:9090/targets#pool-k8s-status
	
![](/media//202106/2021-06-08105614549789.png)

---
##监控cAdvisor（容器监控）
	cAdvisor作为kubelet内置的一部分程序可以直接使用。
	
##方式一：
####k8s默认开通的是https安全端口6443访问。（安全访问）
####添加master的host与端口信息
	vim /etc/profile
	export KUBERNETES_SERVICE_HOST=172.16.16.168
	export KUBERNETES_SERVICE_PORT=6443
	source /etc/profile	#生效

---
####下载地址：[prometheus-rbac.yml](http://baowork.frp.bao-server.club:22080/KODExplorer/index.php?share/file&user=1&sid=3Fe52rWw "prometheus-radc.yml")
	#在k8s master中执行
	kubectl apply -f prometheus-rbac.yml

---
####查看secret
####具体详细介绍可跳转：[k8s权限管理](http://baodoc.frp.bao-server.club:22080/project/35/441/ "k8s权限管理")
	kubectl get secret -A    #查看所有secret
	kubectl get secret prometheus-token-t47mc -o yaml    #查看指定secret

![](/media//202107/2021-07-02142311706378.png)

---
####将上面的数据使用[Base64加密解密在线工具](https://tool.lu/encdec/ "Base64加密解密在线工具")解密后，分别写入
	/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
	/var/run/secrets/kubernetes.io/serviceaccount/token

---
####修改prometheus配置
	vim prometheus.yml
	#内容：
      - job_name: 'kubernetes-cadvisor'
        kubernetes_sd_configs:
        - role: node
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
        - source_labels: [__meta_kubernetes_node_name]
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
        - source_labels: [__address__]
          regex: '(.*):10250'
          replacement: '172.16.16.168:6443'
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        metric_relabel_configs:
        - source_labels: [instance]
          separator: ;
          regex: (.+)
          target_label: node
          replacement: $1
          action: replace
    
        - source_labels: [pod_name]
          separator: ;
          regex: (.+)
          target_label: pod
          replacement: $1
          action: replace
        - source_labels: [container_name]
          separator: ;
          regex: (.+)
          target_label: container
          replacement: $1
          action: replace

---
##方式二：
####开通k8s的非安全端口。
	#开放API server的非安全端口8080：
	#修改k8s api的配置文件
	vim /etc/kubernetes/manifests/kube-apiserver.yaml
	#内容：
	    - --insecure-port=8080   #修改
	    - --insecure-bind-address=0.0.0.0  #新增
	#重启服务
	systemctl daemon-reload
	systemctl restart kubelet

###修改prometheus配置
	vim prometheus.yml
	#内容：
      - job_name: 'k8s-cadvisor'
        metrics_path: /metrics/cadvisor
        kubernetes_sd_configs:
        - api_server: 'http://172.16.16.168:8080'
          role: node
        relabel_configs:
        - source_labels: [__meta_kubernetes_node_name]
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
        - source_labels: [__address__]
          regex: '(.*):10250'
          replacement: '172.16.16.168:8080'
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        metric_relabel_configs:
        - source_labels: [instance]
          separator: ;
          regex: (.+)
          target_label: node
          replacement: $1
          action: replace
        - source_labels: [pod_name]
          separator: ;
          regex: (.+)
          target_label: pod
          replacement: $1
          action: replace
        - source_labels: [container_name]
          separator: ;
          regex: (.+)
          target_label: container
          replacement: $1
          action: replace

###查看
	http://192.168.110.151:9090/targets#pool-k8s-status
![](/media//202106/2021-06-08110402420377.png)

---
##grafana添加k8s监控模板

![](/media//202106/2021-06-08110523769792.png)

![](/media//202106/2021-06-08110536580112.png)

![](/media//202106/2021-06-08110606000873.png)

![](/media//202106/2021-06-08110654910432.png)

---
#以下为额外知识，无需操作：
##设置API server接收所有主机的请求：
	kubectl proxy --address='0.0.0.0'  --accept-hosts='^*$'