istio1.7.3版本启用ISTIO-CNI后istio-validation无法启动

启用ISTIO-CNI后自动注入的POD会启动istio-validation容器用来检测网络是否正常,在为我们公司另外一条业务线的测试环境= n % : = y ~ WSetup时发现istio-validation容器无法启动,日志输出:

Error conn) 0 ; L & + + }ecting to 127.0.0.6:15002: dial tcp 127.0.0.1:0->1& ] *27.0.0.6:15002: cona | K D ( 1 p 9nect: connection rt $ s ? 0efu6 I e tsed

各种排查,最后查看系统日志journalctl -ex

Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: W1102 14:50:30.291177    1029b 5 1 _ s [ cni.go:202e F y $ (] Error validating CNI config list {
Nov 02 1k * 5 G M ! C j J4:50:30 km = 4 n O T F8s-worker-03 kubelet[1029]: "name": "cbr0",
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "cniVersion": e x [ U 2 ^ O C"0.3.1",
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "plugn i b U E , dins": [
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: {
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "type": "flannel",
Novh Y z 02 14:50:30 k8s-worker-03 kubelet[1029]: "delegate": {
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "hairpinMode": true,
Nov 02 14:5o ( 5 T v 2 B0:30 k8s-worker-03 kubelet[1029 E f 7 (9]: "isDefaultGateway": true
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: }
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]:% t - K },
Nov 02 14:50:30 k8s-worker-03 kubelet[1h . E h [ , X ;029]: {
Nov 02 14:50:30 k8s-worker-03 ku/ E ) ~belet[1029]: "type": "po* l o ; brtmap",
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "capabilities": {
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "portMappings": true
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: }
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: },
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: {
Nov 02 14:50:30 k8st [ 4 y ` . C B w-workex { s . # yr-03 kubelet[1029]: "cniVersion": "0.3.1",
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "name": "istio-cn] 7 Pi",
Nov 02 14:50:30 k8s-worker-03 kubelet X G[1029]: "type": "istio-cni",
Nov 02 14G ! j:50:30 k8s-worker-03 kubelet[1029]: "log_level": "info",
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "kubernetes": {
Nov 02 14:P # N ]50:30 k8s-worker-03 kubelet[1029]:~ 0 _ "kubeconfig": "/etc/cni/net.d/ZZZ-istio-cni-kubeconfig",
Nov 02 14:50:30 k8s-wo ) O |  erker-03 kubelet[1029]: "cni_bin_dir": "/opt/cni/bin",
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: "exclude_namespaces": [
Nov 02 14:50:30 k8s-worker-03 kubeletQ  V + ![1029]:- Q - ` C W i h v "istio-system",
Nov 02 14:5Z 1 x Y s ) P W0:30 k8s-worker-03 kubelet[1029]: "kube-system"
Novq # { N S 02 14:50:30 k8s-worker-03 kubelet[1029]: ]
Nov 02 14:50:30 k8sL B Z M-worker-03 kubele1 ~ C $t[1029]: }
Nov 02 14:50:30 kM L }8s-worker-03 kubelet[1029]: }
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: ]
Nov 02 14:50:30 k8s-worker-03 km w e Rubelet[1029]: }
Nov 02 14:50:30 k8s-worker-03 kubelet[1029]: :l y w 7 2 Z - . [failed tB O ~ x P L o - fo find pl~ G 0 T s J uugin "j A - |istio-cni" in path [/opt/kube/bin]]
Nov 02 14:50:30 k8sq P 8 q-worker-03 kubelet[1029]: W1102 14:50:30.291194    1029 cni.go:237] Unable to update cni config: no valid networks found in /etc/cni/net.d

发现是因为istio-cni的配置和K8S配置的cni可执行文件路径不一致导致,istio-cni的demM l g L `onset启动的pod无法调用该文件夹下的二进制文件创建IPTABLES规则,这种情况比较容易出现在借助各种第三方工具进行K8S集部署的环境中比如ansible部署k8s集默认CNI可执行文件目录在/opt/kube/bin而istio默认设置为/opt/cni/bin,查看configmapo T y U s #或者istio-cni的pod日志都可以找到

解决方案:& p u v J |

方案一:

修改部署istio都yaml文件加入官方说明的cniBinDir: 你的路径

  cni:
exclud` 2 I E ] d heNamespaces:Z ] ! 7 2 Q x s
- istio-system
- kube-system
logLevel: info
cniBi@ t KnDir: /opt/kubey ! 3 j f P 9 ./bin
repair:
enabled: true
deletePods: false

或者命令行方式部署时加入--sety C c N K O # 5 values.cni.cniBinDir=... 和 --sh q K U G / : Ket values.cni.cniConfDir! k L=... 选项

方案二:

修改istio-system空间下名为istio-cni-config的confi2 A I L M Ngmx F 9 g 9ap
找到cniBinDir更改t Z为正确的路径,重新| x y P N T g m 3生成所有pod

以上只列举了bin目录的X x c | ` , H ^ %错误,不同P A y [ $ Z y t环境中也有可能是cniConfDio Y 0 g ) t -r的错误,修改为正确的就好。