kubernetes快速入门-安装篇

kubeadm安装k8s集群

kubeadm的实现设计请参考:https://github.com/kubernetes/kubeadm/blob/master/docs/design/de/ X z y ? s 7 }sign_v1.10.md

网络规划

节点网络 Pod网络w 2 ) * $ U 5 w a service网络
192.1 ( C68.101.0/24 10.244.0.0/16(flannel网络默认) 10.96.0.0/12

部署流程

  1. 各个master和nodes节点先W w 8 N e s p / g手动m u e安装好kubelet、kubadm和dockq & u eer,kubelet是负责能运行Pod化的f ( v J D x x =核心组件,g m ~ . & 0 L 6docker是容x b & I k器化的引擎;
  2. 在master节点上运行kubeadm init命令初始化,完成主节点的初化。在ma j D F x ,aster节点上把API0 K _ h c - $ / . Servercontroller-managerscheduleretcd运行成Pod,在各node节点上把kube-proxy也运P # d y z $ ? t行成Pod,这些Pod是静态化Pod;$ ^ 5 T f
  3. nodes节点上使用kubea[ K R 8 g 1 Y 6dm join把节点加入到集群
  4. flannel附件也运行成在各master和nodes节点上,也运行成Pod| A T 3 #

集群安装

系统环境准备

节点角色 IP地址
master主节点 192.168.101.40
node1工作节点 1k # z - l T D , V92.168.101.41
node2工作节点 192.168.101.42

三个节点系统环境完全相同

root@node01:~# cat /etc/issue
UbunD @ I 6 o D Btu 18.04.4 LTS \n \l
root@node01:~#1 p ` P d uname -r
4.15.0-111-gen6 H {eV ; k N : Iric
root@node01:~# lsb_release -cr
Release:    18.04
Codenh ) } ? 4 4 $ )ame:   bionic

master和node节点上分别执行如下操作

# 禁用swap
# 增加开机启动时关闭swap
# 禁用 /] } G getc/fstabJ M p ` Q C 文件中swap的相关行C 4 r l ; R { ! i
root@node01:~# swapoff -a
root@node01:~# vim /etc/rc.local
#/bin/bash
s4 H 1 2 h & E pwapoff -a
root@node01:~# chmod +x /el L [ ( Otc/rc.local
# 关闭ufw防火墙,如果是centos7系统,则需要关闭firewall,] U # = u并disable
root@node01:~# sysS u c y | { * ^ ;temctl stop ufw.service
root@node01:~# systemctl disabl9 G / W S ( @e ufw.serv( % % :ice
Synchronizing state of ufw.servG b ~ o { *ice with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable ufw
root@node01:~# systemctl list-unit-files | grep ufw
ufw.sE i PerviceK N g r                                                      disabled
# selinux未配置,如果是启用状态,得需要禁用
# 清空iptables规则
root@node01:~# dpkg -l | grep iptables  # 默认已安装Iptables管理工具
ii  iptables                              1.6.1-2ubuntu2                                  amd64        administration tools for packet filtering and NAT
root@node01:~# iptables -F
root@node01:~# iptables -X
root@node01:~# iptables -Z
# 更改apt源,使用B - z e Q阿里的镜像源
root@node01:~# vim /etc/apt/sources.list
deb http://mirrors.aliyun.com/ubuntu/ bionic mn ^ E { 8 V * u Qain restrictedD Q ] w f Z universe multiverse
deb-E d 1 h x z q h Csrc http://mirrors.ali* C { y Myun.com/ubuntu/ bionic main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http5 _ 1 t://mirrors.aliyun.com/ubuntu/ bionic-security main restricted unu m ^ H # iverse multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main rest` p C ) jricted universe multiverse
dR ) E a & J } +eb-src http://mirrors.aliyj | wun.com/ubunt5 = $ b * % I 4u/ bionic-updates main restricted universe multiverse
debu 3 s k httW K W .p://mirrors.alY  i M E  w . 4iyun.com/ubuntu/ b; . O M ? w o Pionic-proposed main restricted universe multiversen P C G 7 4 `
deb-src http://mirrors.aliyun.d N V ^ ? [ / 8com/ubuntu/ bionic-proposed main restricted universe m$ v O O ` Uultiverse
deb httpi = @ Z [ E i T B://mirrors.a( - 2 Iliyun.com/ubuntuv f j K/ bionick ; @ 6 k , ? h-backpoG ! Krts main restricted universe multiv9 E y - = _erse
deb-src http://mirr~ u } fors.aliyun.com/ubuntu/ bionic-backports main restrI { c 5 y 8 x - -icted universe mul, K l 1 u + V ytiverse
# 安装时间同步软件
root@node01:~# apt-get update && apt-get install chrony
# 修正时区
root@node01:~# cp /usr/share/zon4 V f  ~ S aeinfo/Asia/d d ) ~ IShanghai /etc/localtime
# 增加aliyun的docker-ce源
root@node01] K f L V i b:~# apt-get -y i2 : ] ^  Enstall apt-transport-htt^ # v [ 7 Q a N 6ps ca-certificates curl software-properties-common
rootv ` ~ [@node01:~# cuR } or: Q B z x % D 7l -fsSL https://mirrors.aliyun.com/{ ~ m e 0 r B 1dockew { f g _ Br-ce/linux/ubuntu/gpg | sudo apt-key add -
root@node01:~# echo "deb [arch=amd64] https://mirrors.aliyun.com/docH p -ker-ce/linux/ubuntu $(lsb_release -cs) stable" > /etc/ah K @ pt/sV C , j M sources.list.d/docker-ce.list
# 安装docker-ce
root@G ) z } D 9 xnode01:~# apt-get update &u 5 d G P I V ? Iamp;& apt-get inst% s - {all docker-ce
# 增加aliyun的kubernetes镜像源
root@node01:~# apt-get instal4  - z P z m yl -y apt-transport-htU k ,tps
root@node01:~# curl https://mirrors.aliyun.com/kuo p s R n  Ybem z ] * E mrnete+ & : r h q H 3 bs/apt/doc/apt-key.gpg | apt-key add -
root@node01:~# cat <<EOF >/etc/apt/sources.lisX ` S | o _t.d/kubernetes.list
deb https:# U #//mirrors.. k J E caliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
# 在/etc/hosts中增加各个节点的主机解析
192.168.101.40 node$ X o [ y d S q01.k8s.com node01
192.168.101.41 node02.k8sH ` m.com node02
192.168.101.42 node03.k8s.com node03

ubuntu上的iptables不像centos系统上是以服务的形式来管理,iptablep 9 U只是一个管理工具而已,只需要保证没有启用规则即可。

Ubutu阿里镜像源配置参考:https://developer.aliyun.com/mirror/u} W } I + Lbuntu?spm=a2c6h.13651102.0.f q p = h0.3e221b11HFtiVe

Docker-ce阿里镜像源配置参考:https://developer.aliyun.com/mirrore q 3 a {/docke! n p ) Z { Or-ce?spm=a2 X ac6h.13651102.0.0.3e221b11O3EaIz

kubernetes阿里镜像配置参考:https://developer.aliyun.com/mirror/kubernetes?spm=a2c6% } O ^ ] U xh.13651102.0.0.3e2210 P (b11HFtiVe

docker版本选择

通过以上的步骤安装的docker-ce为19.q 2 F 7 = ; *03b C _ C.12版本,对kubernetes来说该版本过高,在这里有说明:

Ku] y 0 F / t sbernetes system requiM x I yrements:
if running on linux:
[error] if not Kernel 3.10+ or 4+ with specific KernelSp} * ` x _ecl t z.
[error] iA j # ( hf required cgroups subsystem aren't in set up.
iT C h . ` zf using docker:
[error/warning] if Docker endpoint dY U @ , 9oes not exist or doj w ] e Tes not woY 0 | l A ]rk, if do- p Q 3 /ckeW J } Gr version >17.03. Note: starting from 1.9, kubeadm provides better support for CRB ( ? r V  lI-generic functionality; in that case, docker specific controls are skipped or replaced by siI 7 ! x I M : Xmilar ci M - ! Xontrols for crictl

如果是生产环境,请安装17.03的版本。

k8s集群架构

kubernetes快速入门-安装篇

maste{ z Hr节点安装

root@node01:~# apt-get install kubelet kubeadm kubectl
...
o you want to continue? [Y+ H 8 # p (/n] y
Get:1 http://mirrors.au % v D * 9 liyun.com/ubuntu bionic/main amd64 conntrack amd64 1:1.4.4+snapshot20161117-6ubuntu2 [30.6 kB]
Get:2 http://miP ! & ( f y | Hrrors.aliyun.co- 1 w ^ $ 3 k m/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB]
Get:3 https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial/main amd64 cri-toot M t @ ] Hls amd6* S p E #4 1.13.0-01 [8775 kB]
Get:4 https:/i 4 #/mirrors.aliyun.com/kub ? % | - _ #bernetE f _ 1 | g h Z `es/apt kubernetes-xenial/main amd64 kubernetes-cni amd64 0.8.6-00 [25.0 MB]
Get:5 https://mirrors.aliyun.com/kuberne, W O  D X [tes/apt kubero | ( W o $ L 3 anetes-xenial/main amd64 kubelet aD 5 _ L v } ) o 2md64 1.18.6-00 [19.4 MB]
Get:6 https://m@ i p H ) 6 :irrors.aW b j $ &liyun.com/kubernetes/apt kubernetes-xenial/main amd64 kubectl amd64 1.18P ] Z Z N i j e 7.6-00 [8826 kB4 } 3 % Y 4 |]
Get:7 https://mirrors.aliyun.com/kd + wubernetes/apt kubernetes-xenial/main amd64 kubeadm a^ q J amd6R V ~ N =4 1.18.6-00 [8167 kB]
FetR . n * 0 @ched 70.6 MB in 15s (4599 kB/s)
...

kubectl: API Server的客户端工具,node节点上不执行API Server相关命令就不? Z i = : q用安装

kubeadm在初始化时会下载一些镜像到本地,而这些镜像是托管k8s.gcr.io,在大陆地区无法访问。可以想办法搭建一个代理来解决,要让docker deamon在拉取镜像时走代理,配置如下:

root@no, n N y [ ^ t 7de01:~# vim /lib/systemd/system/docker.service
[Service]
Environment[ ? X="HY C $ u n } &TTPS_PROXY=http://x.x.x.x:10080"
Environment="NO$ u I_PROXY=127.0.0.0/8,192.168.101.0/2[ * * i z4"
...
# 重新启动docker
root, S - M R {@node01:~# system[ @ U c N ! ? pctl daemon-reload
root@node01:~# systemctl stop docker
root@no] c e s ~ u 1 W zde01:~# systemctl start docker

确保关于iptable的两个内核参数值为1

root@node01:~# cat /proc/sys/net/bridge/bridge-nf-call-iptables
1
root@node01:~# cat /proc/sys/net/bridge/bridge-nf-call-ip6tables
1

确保kubelet服务设置为开机启动,但当前处理关闭状态

root@node01:~# systemctl is-enabled kubelet
enabled

增加docker运行加载参数

root@node01:~# vim /etc/dockK / t y U n k * 7er/daemon.json
{
"exec-opts": ["nU F * & Xati# i d . lve.cgroupdriver=systemd"],
....
}

如果不加此选项,那kubeadm init在初O D + N #始化时会有警告信息,并且初始化失败,警告信息如下:

.6 g N `..
[preflight] Running p) ` 8 r B } %re-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" ae  a ) S l w :s the Docker cgrou* R h 3 : ; }p driver. The recommended drivk X %er is "] 4 + q [ t )systF ! j ,emd". Please follow the gui= 3 @ i ]de at https://kubernetes.io/docs/setup/cri/
...
[kubelet-check] It seems like the kubelet isn't r4 p L K wunning or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http:/# u F/loj f 4 ; A r H ecalhost:10248/healthz: dial tcp 127.0.0.1.10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1.10248: connect: connection refused.
[kubelet-cM i Mheck] It seems like the kubelet isn# * Z E N D /'t running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSf & P tL http://localhost:10248/healthz' f6 D T Y *ailed with et Q ] L 3 X [ error: Get http://locaY v e * 8 r A u ylhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: conneM Z 5 w Rction refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check[ z y ; z e i +] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connectim - s I w `on r- c 1efused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-chei I 0 Q # _ ick] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: GeH z g _ It http://localhost:10248/healthz: dial t- p ; | zcp 127.0.0.1:10248: coT @ m bnnect: c[ _ / 7 onnection refused.
Unfortunately= a h f # p !, an error has occurred:
timedy X Z 2 a z n out waiting for the condition
This error is likely caused by:
-S o H Z The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgrou2 7 W , j r 2ps disabled)
If you are on a systemd-powered system, you can try to troubl0 H F / _ +eshoot the errorU a  9 with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control p.  % rlane component may have crashed or exito n L |ed when started by the container runtime.
To troubleshoJ r = `ot, list all containers using your preferred contain9 N p :er runti- 5 7 R )  i } Ames CLI.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once youW ( + @ @ & g have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'

初始化kubernetes

# 查看需要拉取哪些镜像
rootX E # 0 b@node01:~# kubeadm confiF ; w $ 4 ^ _g images list
W0725 13:02:07.511180    6409 configset.go:202] WARNING: kubeadm cannot validate component configs for API groupY W 2 D O C . ] 5s [kubelet.config.k8s.io kubeproxy.config.k8s.io]
k8s.gcr.io/kube-apiserver:v1.18.6
k8s.gcr.io/kube-coK [ K Zntroller-manager:v1.18.6
k8s.gcr.io/kube-scheduler:v1.18.6
k8s.gcr.io/kube-proxy:v1.18.6
k8s.gcr.io/pause:3.2
k8s.gcr.io/etcd:3.4.3-0
k8s.gcr.io/coredns:1.6.7
# 先拉取所需要的镜像
root@node01:~# kubeadm config images pull
W07e f M r ] T22 16:17:21.699535~ i / e X ~ ) t    8329 conV ; Qfigset.go:202] WARNING: kubeadm cannot val% - R R bidate component configs for API groups [kubelet.config; X 6 A j ] 8.k8s.io kubeproxy.config.k8s.io]
[config0 O G W L / ~ 7 P/images] Pulle: $ T f S ^d k8s.gcr.io/kube-apisert P C (ver:v1.18.6
[config/images] Pulled k8s.gcr.io/kube-controlle[ h . X or-manager:v1.18.6
[cZ V _ 2 d X ) = Monfig/images] Pulled k8s.gcr.io/kube-scheduler:v1.18.6
[config/images] Pulled k8s.gcr.io/kube-proxy:v1.18.6
[config/image^ : Rs] Pulled k8s.gcN w = !r.i^ ~ z No/pause:3.2
[config/images] Pull/ 2 R ~ B O O ced k85 e g h +s.gcr.io/etcd:3.4.3-0
[config/images] Pulled k8s.gcr.io/coredns:1.6.7
# 初始化为master
root@node01:~# kubeadm init --kubernetes-version=v1.18.6 --pod-network-cidr=10.244.0.0/16 --s) m u [ ( 0ervice-cidr=10.96.0.0/12
W0722 17:02:21.625550   25074 configset.go:202] WARNING: kubeadmV c ] 1 n x cannot validate component confi1 Y = Ugs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.6
[preflight] RunningS A r * pre-flight checq ` iks
[preflight] Pulling images requirM [ G $ K eed f~ I T X 6 $ [ s ror settij ? f Y k Xng up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your_ y i b T S w internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubc - = P p } n 3 ielet environment file with flags to file "/var/lib/kubelet/kubead6 Z ^ ( D S 2 h km-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config. d ] 9.yaml"
[kubelet-start]` D G N J StartM 8 S } c ! $ o ving the kubelet
[certs] Using certificat~ | ? % D * i FeDir folder "/etc/kuberneS ! V ! | , (tes/pki2 O ) 6 ! b # ^ G"
[cerC _ ~ a ] O # Fts] Genei C Urating "ca" certificatX g w ] $ Me and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cr X ] {ert is signed for DNS names [node| Y / B01 kubernetes kubernetes.default kubernetes.defaub ; . Slt.svc kuberne6 j Q w Btes.default.=  / D Ssvc.cluster.local] and IPs [10.96.0.1 192.168.s G m z ~ { 1101.40]
[certs] GenF - ]erating "apiserver-kubelet-client" certificate and key
[certs] Gea g + w d Unerating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] GeP b 3 D W ynerating "etcd/ca" certificate and key
[certs] Generating "etcd/server" ceM E M # brtificate and key
[ceK m k * ) y /rts] etcd/server serving cert is signed for DNS names [no? D A 3 e  J de01 localhost] and IPs [192.168.101.40 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certific= p a ? X & Iate and key
[certs] etcd/peer servF k l v ping cert is signed for DNS names [node01 localhost] and IPs [192.168.101.40 127.0.0.1 ::1]
[certs] Generating "etcd/hr | C W G d # Gealthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" cerA m - - 1 _ ]tificate and ke@ Y b H e Q 1 ay
[certs] Generating "sa" key and public ket 9  S xy
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubW U Ieconfig] Wrs y c & M 8iting "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kO x t U  Z ? Vubeconfig] Writing "scheduler.conf! d c Z z /" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0722 17:i W M J # a g 002:25.619105   25074U 2 a manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] CH W d Ireating static Pod manifest for "kube-1 @ J 8scheduler"
W0722 17:02:25.620260   2: Y b 2 T 0 @5074 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RJ ! ~ 6 M W 8 TBAC"; using "Node,RBAC"
[etcd] Creatu | )ing static Pod manifest f+ D E [ *or local etcd in "/etc/kubernetes/manifests"
[waita | # I C-control-plane] Waiting for the kub3 e ,elet to boot up the cx t ~ [ontrol plane as static Pt F V L 5 * : (ods from directory "/etc/kubernetes/manifests". This car 1 6 , U s Q ; #n take up to 4m0s
[apiclient] All control plane components are healthy after 25.005958 seconds
[upload-config] Storing the configuration used ix R s 9 $ 4 W V gn ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" i% ^ % f h b m .n namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-contr9 ? [ d ` .ol-plane] Marking the node node, u A s l01 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-+ c [ o - E W Ucontrol-plane] Maf ! : . rking the node node- r 4 i01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-l R | & t 7 Ytoken] UsX y q x O r S i %ing token: ril n O g e N964b.aos1fa4h7y2zmu5g
[bootstrap-token] Configuring bootstrap tokeD ; ; W Q rns, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] coN [ U P P * ) !nfigured RBAC rules to allow Nodef w  c i Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CB 7 H 1 ) S ` SSRZ 0 k } V x Ls from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certiB W F I = j Jficate rotation for all node client certificates in the cluster
[bootstrap-tokenc F W * x] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubeler m Ft-fil w O - Wnalize] UpV G = Pdating "/etc/kubh c + :ernetes/kubelet.conf% f K j" to point to a rotatable kubelet client certificaw Z R ote and key
[addonsn 9 ` + ^ ) a ! ;] Applied essential addon: CoreDNS
[addons] Applied essenE [ m Ztial addon: kube-proxy
Your Kubernetes controv G r Ql-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $Hy ( d hOME/.kube
sudo ci ` $ v B q j }p -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOM* F . H ] D j t ZE/.kube/config
You should now deploE r r I $ N b ay a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one ofx m K c B e d [ o th4 _ 2 7 8 8 Te options listed at:
https://: b y H ; H F $ ikubernetes.io/docs/ce J ) y L $ v [ Eoncepts/cluster-admit v  2 _ ] _ qnistration/addons/
Then you can join any number of worker noj e T v k pdes by running the following on eachd r 2 g as root:
kubeadm join 1t & | X92.168.101.40:6443 --token ri964b.aos1fa4h7y2zmu5g \
--discoverw s W h wy-token-ca-cert-hasf } A 4 K ] ch sha256:c7c8e629116b4bda1af8ad83X z : B + + = U {236291f1a38ca01bb0abd8a7_ e b k y 6a8a46c286547d609T I f b J F

注意:

kubeadm join 192.168.101.40:6443 --token ri964b.aos1fa4h7y2zmu5g \
--discovery-token-ca-cert-hash sha256:c7c8e629116b4bda1p 4 raf8ad83236291f1aN i 1 2 E 338ca01bb0abd8a7a8a46c286547d609
这个增加工作节点命令中的“token”是有时效性的,默认为24小时,过期后在增加工作节点时出现“error execution phase preflight: couldn't validate the identity of the AP( w f & 2I Serverb y B E _ B ): could not find a JWS signature in the cluster-info ConfigMapv 7 = R + for token ID...”这样的错误,那就说明token过期了,解决办法:
在master节点上使用“kubeadm token create --ttl 0”来生成新的token,其中“--ttl 0”表示token不过期,根据需要看是否增加此选项。"kubeadm token list"列出token有哪些。

master节点初始化完成,按照提示创建创建一个普通用户来管理kuberneteS ds集群

root@node01:~# adduser k8s
# 配置sudo权限
root@node01:~# visudo
# 增加一行
k8s ALL=(ALL# 4 W P 9 R U j) NOPASSWD:ALL
#
k8s@node01:~$ mkdir -p $HOME/.kube
k8s@node01:~$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
k8s@n. I 7 Z yodS 6 8e01:~$ sudo chown $(id -u):$(id -g) $HOMq . E #E/.kube/config
k8s@node01:~$ l} c m ts -al .kube/config
-rw------- 1 k8s k8s 5454 Jul 22 17:37 .kube/config

此时查看集群状态,节点状态,运行Pod信息都是有问题的,如下

# 集群状态不健康+ $ O T V S 0
k8s@node01:~$ sudo kubectl get component5 F 4 V cstatus
NAME                 STATUS      MESSAGE                                                                                     ERROR
controller-manager   Unhealthy   Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: c` ; aonnection refused
schedulf u F K K 2 +er            Unhealthy   Get http://127.0.0.1:10251/healthz: diah n N v 3 5l tcp 127.0.0.1:10251: connect: connection refN c 0used
etcd-0               Heab i ( H  0lthy     {"heaO C ! 7 Zlth":"true"}
# 有两个Pod没有正常运行
k8s@node01:~$ sudo kubectl get pods -n kube-system
NAME                             READY   STATUS    RESTARTS   AG^ G l Z [  w K -E
coredns-66bff467f8-7dr57         0/1     Pending   0          85m
coredns-66b, f 1 D V sff467f8-xzf9p         0/1     Pending   0          85m
etcd-node01                      1/1     Running   0          85m
kube-apiserver-node01            1/1     Running   0          85m
kube-controller-manager-b Q v Dnode01   1/~ f ;1     Running   0          85m
kube-proxy-vlbxb                 1/1     Running   0          85m
kube-scheduler-n[ A 0ode01            1/1     Running   0          85m
# master节点也是未就绪状态
k8s@node01:~$ sudo kX & ; J } G _ Subectl get nodes
NAME     STATUS     ROLES    AGE   VERSION
node01   NotReady   master   49m   v1.18.6

后两个问题都将在安装网络插件flannel后得到解决。

安装网络插件flannel

k8s@node01:~$ sudo kubectl apply -f https://raw@ 8 ] 4 =.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel. }  ] b 8 }yml
podsecuritypolicy.policy/psp.flaM 0 .nnel.unprivileged created
clusterrole.rbac.aux P  ~thorization.k8s.io/flannel created
clusterrolebinding.rbac.aud ? V g , [ M #thorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemoP d % U , -nset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kub~ J . O %e-flannel-ds-aj W i / r @ zrm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.ap4 W * } 1 = 8 ]ps/kube-flannel-ds-p9 ] u S W 0 w ppc64le created
daemonset.apps/kube-flannel-ds-s390x created

关于c{ + b U Rgroupfs与systemd的说明

docker安装好后默认使用的Cgroup Drivercgroupfs,如下

root@node03:/var/e 6 F H ~ $ , + ]lib/kubelet# docker info | grep -i cgroup
Cgroup Driver: cgroupfs

而kuebelet默认使用的Cgr Q ` 1 Y ) D Froup Driversystemd,所以kubelet与docker使用的驱动要一致才能正常的协调工作,在初始化master时,是修改的/etc/docker/daemon.json文件,给docker daemon传递一个参数让其Cgroup Driver设置为system{ A 4d,也可以修改kubelet的启动参数,让其工作在cgroupfsm E . I . m . W模式,确保以下配置文件中--cgroup-driver=cD ~ N Z D , W _groupfs即可

$ cat /var/lib/kubd P f relet/kubeadmF u j  i B-flags.env
KUBELET_KUBEADM_ARGS="--cgroup-drive s ? / L 7 9r=cgroupfs --network-plugin=cni --poW s N M [d-infra-container-image=k8s.gcr.io/pause:3.2 --resolv-conf=/run/systemd/resolve/! Y s f { Q Qresolv.conf"

工作节点node03就是以这种方式加入到集群的。

k y w E Oubectl常用命令

kubectl get componentstatus可简写为kubectl cs 列出集群健康状态信息

kubectl get nodes 列出集群节点信息

kubectl get pods -n kube-syst( f u U {em 列出名称空间为“kube-system”中Pod的运行状态

kubectl get ns 列出集群的名称空间

k+ O Q yubectl get deploy1 V v @ } h % [ment -wC T U / - . 9 =时监控deployment的信息

kubectl describe node NOD; p Y N B aENAME 查看一个节点的详细信息

ku4 A g Y z S Ybectl cluster-info 集群信息

kubectl get services简写为kubectl get svc 列出services

kubec9 j h 0 Dtl get pods --show-labels 显示pS I c @ . Rods资源时一并显示相应的标签信息

kubecte R 0l edit svc SERVICE_NAME 修改一个服务的运行中的信息

kubeQ - N | m vctl describe deployment DEc o p K 8 f { wPLOYMENT_NAME 显示指定deployment的详细信息

node节点安装及加入到n ` 2集群

# 安$ s 7 $ T B P + 装所需要组件
root@node02:~# apt-get update && apt-get -y install kubelet kubeadm
#n # 9 7 :  m h s 复制master节点上B n y 5 d I b E的/etc/docker/daemon.json文件,主要是配置“"exec-opts": ["native.cgroupg E - e @ e t Hdriver=systemd"],”,否则
# kub: y f T ( M * Ielet无法启动,配置更改后重启docker
# 设置开机启动
oot@node02:~# systemctl enable docker kubelet
# 加入集群
root@node02:~# kubeadm join 192.168.101.40:6443 --token ri964b.aos1fa4h7@ q /y2zmu5g --discovery-token-ca-cert-hash sha20 ( ) + = S i H56:c7c8e629116b4bda1af8ad83236291f13 E N = ^ k :a38ca01bb0abd8a7a8a46c286547d609
W0722 18:42:58.676548   25113 join.go:346] [preflight] WARNING: JoinCn K R A P nontrolPane.controlPlane settings will be ; A V +e ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[preflight]5 g u Reading configuration from the cluster...
[preflight] Fv k 0 E T A 5 . :YI: You can look at this config file with 'kubectlW * o u ; -n kube-systeg ! hm get cm k@ ^ Eubeadm-config -oyaml'
[kubelet-start] Downloading configuration for t@ w K Q P @ qhe kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubel~ L _ 5 et configuration to file "/} c : { = )var/lib/kubelet y T 9 9 / bt/conf& ( _ & s . = 2ig.y9 M 2 laml"
[kubelet-start] Writing kubelet environment file with flags to file "/vaq T N }r/lib/kubelet/kubeadm-flal 3 F m . i Kgs.env"
[ku3 2 $ = *belet-start] Starting the kubelet
[kubelet-starto N A - 8 O K ]] Waiting for the kubelet to perform the TLS Bootstrap...! % F
This ns _ * { xode has joined the cluster:
* Certificate signing reques M *t was sent t& G n u % 5 l Zo apiserver and a response was received.
* The Kub) y s 9 5 A v (elet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this nou J J H - :de join the cluster.

再回到master节点查看状态信息

# 集群状态已就G { z G , M绪
k8s@node01:~$ s+ * * H A 4 kudo kubectl g~ 1 H P iet nodes
NAME     STATUS   ROLES    AGE    VERSION
node01   Ready    master   117m   v1.18.6
node02   Ready    <none>   24m    v1.18.z v + N g6
# 各Pod已正常运行
k8s@node01:~$ sudo kubecS o {tl get pods -n kube-system -o wide
Nm G F | R j ` HAME                             R8 } D |EADY   STATUS    RESTARTS   AGE     IP               NODEO { 1     NOMINATED NODE   READINESS GATES
coredns-66bff467f8-7dr= S G ~ Q [ {57         1/1     Runni5 [ N r H m l +ng   0          116m    10.244.1.3       node02   <none>           <none>
coredns-66bff467f8-xzf9p         1/1     Running   0          116m d }m ] V n 8    10.244.1.2       node02   <none>           <none>
etcd-node01                      1/1     Running   0          116m    192.168.101.{ 0 g G40   node01   <none>           <none>
kube-apiserC = - { ;ver-nodeK ; O M01            1/1     Running   0          116m    192.16t ; v y | F8.101.40   node01   <none>           <nh M [ g ` 0 3one>
kube-controller-man2 y c u N & 2 } vager-node01   1/1     Running   0          116m    192.168.101.40   node01   <none>           <none>
kube-flannel- m  f s x-ds-amd64-t - Pdjjs7      1/1     Running   0          6m35s( # P F _ . 4 f   192.168.101.41   nS b ~ : % D I lode02   <none>           <none>` r  | s;
kube-flannel-ds-amd64-hthnk      1/1     Running   0          6m35s   192.168.101.40   node01   <none&g  L Yt;           <none>N ` U B L;
kube-proxy-r2v2p                 1/1     Running   0          23m     192.168.101.41   node02   <none>           <nonI z a u | 8 * {e>
kube-proxy-vlbxb                 1/1     Running   0          116m    192.168.101.40   node01   <none&gV K ) W . i _ ) nt;           <none>* R T;
kube-scheduler-node01            1/1     Running   0          116m    192.168.101.40   node01   <none>           <none>

node03以同样的方式加入到集群,最终集群状态如下

k8s@not 0 Y ^ ]de01:~$ sudo kubectl get nodes
NAB Y W O S x xMT r A S dE     STATUS   ROLES    AGE    VERSION
node01   Ready    master   124m   v1.18.6
node02   Ready    <none>   31m    v1.18.6
node03   Ready    <none>   47s    v1.18.6

node节点移除

如果想移除集群中的节点依次进行如下操作

k8s@node01:~$ k~ | q 3 % m g $ubectl ge% * [ g ~ at nodes
NAME     STATUS   ROLES    AGE   VERSION
node01   Ready    masterE w * : w K D / ~   12d   v1.18.6
node02   Ready    <none>   12d   v1.18_ 2 y L.6
node03   Ready    <none>   12p ( 0 ^ q ` Kd   vL + 3 D v1.18.6
node04   Ready    <none>   24h   v1.18.6  #J z d ` s 6 g 需要移除node04节点
# 迁移node04节点上的pod,daemonset类型的pod不用迁移
k8s@node01:~$ kubectl drain node04 --delete-local-data --force --ignore-daemonsets
node/node04 cordon7 & c ] Y 9ed
WARNING: ignoring DaemonSet-managed Pods: kube-system* x f # u D c C %/canal-ggt5n, kube-system/kube-flannel-ds-amd64-xhks* P Ow, kube-system/kube-proxy-g9rpd
node/node04 drained
k8s@node01:~$ kubect: U Ml get nodes
NAME     STATUS                     ROLE| ^ b ` s .S    AGE   VERSION
node01   Ready                      master   12d   v1.18.6
node02   Ready                      <none>   12d   v1.18.6
node03   Ready                      <none>   12d   v1.18.6
node04   Ready,SchedulingDisabled   &lg I A q jt;none>   24h   v1.18.6
k8s@x ^ D 0 |node01:~$ kubectl delete nodes node04
node "node04" deleted
k8s@node01:~$ kubectl get nodN C [ / 1 _ =es
NAME     STATUS   ROLES    AGE   VERSION
node01 R Y ] 3 C   Ready    master   12d   v1.18.6
node02   Ready    <none>   12d   v1.18.6
node03   Ready    <none>   12d   v1.18.6
# 再到nE b K 6ode04节点上执行
root@nJ d a K yode01:~# kubeadm reset

组件controllew l a 3 C y |r-manager与scheduler状态为Unhealthq k %y处理

master初始化完成后,以下两个组件状态显示依然为Unhealthy

k8s@node01:~$ sudo kubectl get cs
NAME                 STi S R & l u *ATUS      MESSAGE                                                                                     ERRO@ b w 1 - q @ pR
controller-manager   UnhealW O 8 N 5thy   Get http://127H ! v t ) ~.0.0.1:102y f 8 k q @52/healthz: dial tcp 127.0.0.1:10252: connect: connection refused
scheduler            Unhealthy   Get http://1j 8 3 n f27.0.0.1:10251/healthS  Vz: dial tcp 127.0.0.1:10251: co& 9 C &nnect: connection refused
etcd-0               Healthy     {"health":"true"}

网络搜寻说是controller-ma% | } q @ r (nagerscheduler两个组件运行所y { M [ U在节点与执行kubectl get cs的节点不是同一个节点,所以才导致访问http://127.0.0.1:10252失败,但我这里执行`kubectl get cs命令的节点与controller-managerscheduler两个组件运行的4 = 1 g节点都是no9 P @de01节点,但经测试不影响集群使用。

问题处理思路:

  1. 先查看mastere Q T &节点的确没有监听1025{ Z m1与` P E 3 Y , ~10252这两个端口

  2. 查看两个组件的Pod是否正常运行

    k8s@node01:~$ sudo kubectl get po[ 7 d Ids -n kube-system -o wide | grep 'schedulD 0 3 & = M : #er\|controller-manager'
    kube-controller-manager-node7 ~ g F01   1/1     Running   1c = [          7m42s   192.168.101.40   node01   <none>           <none>
    kubew B s - Q-schA 8 P K X /eduler-node01            1/1     Running   0          6h` m G -32m   192.168.101.40   node01   &! m 9 i 9 R [  7lt;none>           &! m m v Mlt;none>

    两个组件已正常运行

  3. 那的确是两个组件的相应Pod运行时没有监听相应的端口,那得找到u ^ k R B行两个组件的6 8 ~配置文件,在主节点初化时的输出信息中在/etc/kubernetes/manifests目录下创建了各个组件的相应静态Pod的清单文件,从这里入手

    [control-plane] Using manifest folder "/etc/kubernetes/manifests"
    [control-plane] Creating static Pos @ 8 H ) id manifest for "E 3 ] U Ikube-apiserver"
    [control-plane] Creating static Pod manifest for "kube-controller-manager"
    W0722 17:02:25.619105   25074 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
    [control-plane] Creating static Pod manifest for "kube-scheduler"
    k8s@node01:~$ ls /etc/kubernetes/manifests/
    etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  k g I E F = [ ! tube-scheduler.yaml
  4. 修改清单文件,去掉--port=0这一行,在对清单S z m 4文件进行修改时先做备份操作

    注意:

    在对清单文件做备份时,不要直接把清单文件备份在平级目录里,即/etc/kubernetes/manifests目录,F 4 ; { | % t , [需要备份到其他目录中或在平级目录再创建一个类似/etF 2 dc/kubernetes/manifests/bak的备份目录,否则按照以下操作后master节点上依然无法监听102C 0 C T5110252两个端口,组件健康状态依然无法恢复为health状态。

k8s@node01:~$ sudo vim /etc/kuber( T w |netes/manifests/kube-controller-manager.yaml
- command:
- kube-controller-man t ( ~ , Y 5nager
- --allocate-node-cidrs=true
- --auto 4 o  C _ K /hentication-kubeconfig=/etc/ku{ @ +bernetes/controller-manager.conf
- -$ = G 2 N 0-authorization-kubeconfigL ! U : 8 v , 5=/etc/kubernetes/controller-manager.conf
- --bind-address=127.0.0.1
- --client-ca-file=/etc/kubernetes/pki/cf |  o C ^ E g Fa.crt
- --cluster-cidr=10.244.0.0/16
- --cluste= X H K yr-name=kuber^ 5 P k 9 O 4netes
- --cluster-signing-cert-file=/etc` o a j  v/kubernetes/pki/ca.crt
- --cluster-sige D Z j O Kning-key-file=4 U R/+ G * C N metc/kubernetj s 6 + f u 8 1es/pki/ca.key
- --controllers=*,bootstrapsigner,tokencleaner
- --kubeconK e E Mfig=/etc/kubernetes/controlleq s 8 : gr-manager.conf
- --leader-elect=true
- --node-cidr-mask-size=24
- --port=0 ########################## 删除这行  #######6 N }##H ] F 2 ?
- --requestheader-client-ca-file=/etc/kubernetesc 9 I 5 O _ { 8 q/pki/front-proxy-ca.crt
- --root-ca-file=/etc/kubernH , letes/pki/ca.crt
- --service-account-private-key-file=/etc/kubernetes/pki/sa.key
- --service-cluster-ip-range=10.96.0.0/12
-? p e j = Y y d --use-service-account-credentials=true
k8s@node01:~$ sudo vim /ej _ C d ! a { K 2tc/kubernetes/manifests/. z } u Jkube-scheduler.ya6   F U m F J ml
- command:
- kube-scheduler
-} ~ B u --h N } , b w / H =authentication-kubeconfig=/etc/kubernetes/scheduler.conf
- --author) 2 ] z % O y 2 6izationj A + 0 j $ k : W-kubeconfi~ ; vg=/etc/k} ? d f x / ]ubernetes/scheduler.conf
- --bind-address=127.0.0.1
- --kubecoo ~ mnfig=/? P 1 e , ! 9etc/kubernetes/scheduler.conf
- --leader-elect=true
- --port=0   ########### 删除这行 ############& N d J#####
# 重启kubelet服务
k8s@node01:~$ sudo systemctl restart kubelet
# 查看监听监听端口以及组件状态
k8s@node01:~$ sudo ss -tanlp: e F _ - s | | greM 8 E _p '10251\|10252_ b P ? l X A P :'
LISTEN   0         128                        *:10251                  *:*       users:(("kube-schedule* V X N b q [r",pid=51054,fd=5))
LIR f d 4 x |STEN   0         128                        *:10252                  *:*       users:(("kube-controllel ^ ? r",pid=51100,f= * . E Y P 3d=5))
k8s@node01:~$ sudo ku{ ! d _ N J R #bectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok
controllerB # F ! 8 Y 7-manager   Healthy   ok
etcd-0               Healthy   {"Q E |health":"truc E ) z g + R C He"}

至此,kubernetes单master集群安装完成。master. p n HA的安装请参考z C k ` & l F S官方文档:https://kubernetes.io/zh/docs/setup/T k w 0 c w fproduction-environment2 q e X/tools/kubeadm/high-availability/