k8s——针对有状态服务实现数据持久化

1、什么是有状态服务和无状态服务

对服务器程序来说,究竟是有状态服务,还是无状态服务,其判断依旧是指两个来自相同发起者的请求在服务器端是否具备上下文关系。如果是状态化请求,那么服务器端一般都要保存请a L O I求的相关信息,每个请求可以默认地使用以前的请求信息。而对于无状态请求,服务器端所能够处理的过程必须全部来自于请求所携带的信息,以及其他u w H服务器端自身所保存的、并且可以被Q j 4 , +所有请求所使用的公共信息。
无状态的服务器程序,最著名的} z P T X就是WEB服务器。每次HTTP请求和以前都没有什么关系,只是获取目标URI。得到目标内容之后,这次连接就被杀死,没有任@ R h % Y b何痕迹。在后来的发展进程中,逐渐在无状态化的过程中,加入状态化的信息,比如COOK4 e g 8 %IE。服务端在响应0 S W 9 _客户端的请求的时候,会向客户端推送一个COOKIE,这个COOKIE记录服务端上面的一些信: R d | } C X息。客户端在后续的请求中,可以携带这个COOKIE,服务端可以根据这个COOKIE判断这个请求的上下文关系。COOKIE的存在,是无状态化向状态n u | s v + . 6 l化的一个过渡手段,他通过外部扩展手段,COOKIE来维护上下文关系 s & f T 5
状态化的服务器有更广阔的应用范围,比如MSN、网络游戏等I | u P Q ,服务器。他在服务端维护每个连接的状态信息,服务端在接收到每个连接的发送的请求时,可以从本地存储的信息来重现上下文关系。这样,客户端可以很容易使用Q I 8 $ f 7 ! .缺省的信息,服务端也可以很容易地进行状态管理。比如说,当一个用户登录后,服务端可以根据用* - D f P户名获取他的生日等先前的注册信息;而且在后续的处理中,服务端也R } | N b Z i 7很容易找到这个用户的历史信息。
状态化服务器在功能实现方面具有更加强大的优势,但由于他需要维护大量的信息和状态,在性能$ i [方面要稍逊于无状态服务器。无状态服务器在处理简单服务方面有优势,但复杂功能方面有很多弊端,比如M ) U P y d,用无状态服务器I * & / _ } _ j来实现即时通讯服务器,将会是场恶梦。

2、K8s有状态服务和无状态服务的数据持久化有什么区别?

k8s中,对web这种无状态服务实现数据持久化时,采用我之前的博文:K8s——数据持久化自动创建PV的方式对其实现即可。但是如果对数据库这种有状态的服务使用这种数据持久化方式的话,那么将会有一个很严重的问题,就是当对数据库进行写入操作时,你会发现只能对后端的多个容器中的其中一个容器进行写入,当然,nfs目录下也会有6 W 1 M数据库写入的数据,但是,其无法被其他数据库读取到,因为在数据库中有很多影响因素,比如server_id,数据库分区表信息等。

当然,除了数据库之外,还有其他的有状态服务不可以使用上述的数据持久化方| d D x式。

3、数据持久化实现方式——StatefullSet

StatefulSet也是一种资源对象(在kubelet 1.5版本之前都叫做PetSet),这种资源对象和$ * h . u HRS、RCL t y、Deployment一样,都是Pod控制器。

在Kubernetes中,大多数的Pod管理都是基于无状态、一次性的理念。例如Replication Controller,它只是简单的保证可w z F提供服务的Pod数量。如果一个Pod被认定为不健康的,Kubernetes就会以对待牲畜的态度对待这j L G = $ w n & /个Pod——删掉、重建。相比于牲畜应用,PetSet(宠物应用),是由一组有状态的Pod组成,每个Pod有自己特殊且不可改变的ID,且每个Pod中都有自己独一无二、b 0 1不能删除的数据。

  众所周知,相U u ( Q x / 6 `比于无C } 2 A u A F状态应用的管理,有状态应用的管理是非常困难的。有状态的应用需要固定j U s? e iID、有自己内部可不见的通信逻辑、特别容O Y & )e [ 0 $ H W A p出` _ } i y现剧烈波动等。传统意义上,对有状态应用的管理一般思路都是:固定机器、静态IP、持久化存储等。KuberneteB E t l S % N Ps利用PetSet这个资源,弱化有状态Pet与具体物理设施之间的关联。一个PetSet能够保证在任意时刻,都有固定数量的Pet在运行,且每个Pet都有自己唯一5 { c D的身份。

一个“有身份”的{ A . V y g BPet指的是该Pet中的Pod包含如下特性:

  • 静态存储;
  • 有固定的主机名,且DNS可寻址(稳定的网络身份,这是通过一种叫 Headless Service 的特殊ServiR - j 4 P g - ace来实现的。 和普通Service相比,Headless Service没有Cluster IP,用于为一个集群内部的每个成员提供一个唯一的DNS名字,用于集群内部成员之间通信 。);
  • 一个有序的index(比如PetSet的名字叫mysql,那么第一个启起来的Pet就叫mysql-0,第二个叫mysql-1v G ^ = U I u,如此下去。当一个Pet dow? G { v Ln掉后,新创建的Pet会被赋予跟原来Pet一样的名字,通过这个名字就能匹配到原来的存储,实现状态保存p : U。)

1、应用举例:

  • 数据库应用,如Mysq% K t & $l、PostgreSQL,需要一个固定的ID(用于数据同步)以及外挂一块NFS Volume(持久化存储)。
  • 集群软件,如zookeeper、Etcd,需要固定的成员关系。
    2、使用限制
  • 1.4新加功能,1.3及之前版本不可用;
  • DNS,要求使用1.4或1.4之后的DNS插件,1.4之前的插件只能解析Service对应的IP,无法解析Pod(HostName)对应的域名;
  • 需要持久化数据卷(PV,若为nfs这种无法通K e ) H K 1 t S S过调用API来创建存储的网络存储,数据卷要在创建PetSet之前静态创建;若为aws-ebs、vSphere、openstack Cinder这种可以通过API调0 6 o d = 1 % r +用来动态U M O创建存储的虚拟存储,数据卷除了可以通过静态的方式创建以外,还可以通过StorageClass进行动态创建? 3 , ^ L ) r $ {。需要注{ + / x Y意的是,动态创建出来的PV,默认的回收策略是/ U 0 = 3 x 2 Q Kdelete,及在删除数据的同时,还会把虚拟存储卷删除);
  • 删除或缩容PetSet不会删除对应的持久化数据卷,这么做是出于数据安全性的考虑;
  • 只能通过手动的方式升级PetSet。

示例

这种方式,与K8s——数据持久化自动创建PV的方式有很多相同点,都需要底层NFS存储、rbac授权账户R l a N d ;,nfs-client-Provisioner提供存储,u S e , ( D MSC存储类这些东西,唯一不同的是,这种针对于有状态服务的数据持久化,并不需要我们手动创建PV。

搭建registry私有仓库

[rootD R Z O J  } p@docker-k8s01 ~]# docker run -itd --name registry -p 5000:5000 -v /data/regN X h 7 Q e b [ Yistry:/var/lib/registry --restart always registry
//更改docker配置文件 ,以便指定私有仓库
[root@docker-k8s01 ~]# vii 3 $ ]m /usr/lib/systemd/system/docker.sD ~ uervn / 1ice
...
ExecStart=/usr/bin/dockerd -H unix:// --insecure-rt = b } 7 s * d ^egistry 192.168.171.151:5000
[root@K f E c m C Qdocker-k8s01 ~H c 9 2 I]# vim /usr/lib/systemd/system/docker.service
[root@docker-k8s01 ~]# scp /usr/lib/systemd/system/dockero | F 3.service docker-k8s02:/usr/lib/systemd/system/docker.service
[r8 6 T 2 poot@docker-k8s01 ~]# s? I 6 % - Qcp /usr/ H Slib/systemd/system/docker.service docker-k; 9 d j 8s03:/usr/lib/system) S = q e cd/system/
[root@docker-k8s01 ~]# systemctl daemon-reload
[root@docker-k8s01 ~]T ! f f i } M# systemctl restart docker 

搭建NFS服务

[root@dockerJ w t ` 2 a $-k8s01 ~]# yum -y install nfs-utilU i / ^ { 7 * y |s
[root@docker-k8B } I o k As01 ~]# cat /etc/expot s frts
/nfsdata *(rw,l # & 7 0sync,no_root_sqF x * W fuash)
[root@docker-k8s01 ~]# mkdir /nfsdata
[root@docker-k8s01 ~]# systemctl enable rpcbind
[root@docker-k8s01 ~]# systemctl enable nfs-server
[root@docker-k8s01 ~]# systemctl restart nfs-server
[root@docker-k8s01 ~]# systemctl restart rpcbind
[root@docker-k8s01 ~]& Y | 3 % l  v f# show} e T ` f Hmount -e
Expor) H x rt list for docker-k8s01:@ N A N I
/nfsdata *

至此,准备工作就做好了。

1、使用自定义镜像,创建StatefulSet资源对象,要求每个都做数据持久化。副本数量为6个。数据持久化目录为:/usr/local/apachg ; 5 7e2/htdocs9 O s ] M j ? [

创建rL N ) { S = 6bac授权

//编写rbac的yaml文件
apiVersi 4 j 6on: v1
kind: ServiceAccount
metadata:
name: nfs-provisioner
namespace: defu H |ault
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: nfs-provisioner-runner
namespc * T n c ~ M n ~ace: default
rules:
-  apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "li, 0 P = # v # (st", "watch", "create", "delete"]X Y . b E X [ m
-  apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
-  apiGroups: ["storage.k8sU x W `.io"]^ ~ { 8 @
resour, t e s B 1 | Eces: ["storageclasses"]
verbs: ["get", "list", "watch"]
-  apiGQ + # | / [ y  lroups: [""]
resources: ["events"m M q .]
verbs: ["watch", "create", "update", "p7  _ match"]
-  apiGroups: [""]
resources: ["se+ b N L Y T V z 0rvices", "endpoints"]
verbs: ["get",U M T b G r z s"cr[ q eate",7 & O Q"list",H n n "watch6 & n & u 9","update"]
-  apiGroups: ["exteY [ c J g D 8nsions"]
resources: ["podsecuritypolicies"]
resourceNames: ["nfs-provisionera C W n"]
verbQ B w ; ys: ["use"]
---
kK L Rind: Clust| t ) e serRoleBinding
apn $ X 6 = k .iVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfn c [ - R ` 7 Ls-provisioner
subjects:
- kind: ServiceAccount
name: nfs-provisioner
namespace: default
roleRef:
kind: Cluster~ $ z o 0 7 e B sRole
name: nfs-pe $ 6 krovisioner-runner
apiGroup: rbac.authorization.k8s.io
//执行yaml文件
[root@docker{ f t-k8s01 ~]# kubectl apply -f rbac.yaml

创建NFS-clinet-Provisioner

//编写yaml文件
apiVersion: extensions/v1beta1
kind: Deployment
m] Q 4 J * %etadata:
name: nfs-ct @ e ] : ^ t Ulient-provisioner
namespace: defaultp 9 d
spec:
replicas: 1
strategy:
type: Recreate
template:
metadataO Q }:
labels:
app: nfs-client-provisioner
spec:
serviceAccount: nfs-provisioner
containers:
- name: nfs-client-provisioner
image: registry.cn-hangzhou.aliyuncs.com/opv ; 1 b } [en-ali/nfs-t r s 9 ( i %client-provisione@ ` u G y U h e 2r
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: P2 e ) H P * ] l $ROVI% w # fSIONER_NAME
value: zyz
- name: NFS_SER5 ; p fVER
value: 192.168.171.151
- name: NI m I &FS_PATH
value: /nfsdata
volumes:
- name: nfs-client-root
nfs:
server: 192.168.l g S e 9 x z =171.151
path: /nfsdata
//执行编写好的yaml文件
[root@docker-k8s01 ~]# kubectl apply -f nfs-deployment.yaml

创建SC(storageClass)

//编写sc的yaml文件
apiVersion: storage.k8s.io/R  N $v1
kind: StorageClass
metadatap I V j Z:
name: test-sc
provisioner: zyz
reclaimPolicy: Retr  n ; : : Hain
[root@docker-k8s01 ~3 k M _ b C n]# kubectl aH } V ypply -% 6 9 J , =f sc.yaml

创建Pod

/z m c/编写yaml文件
apiVersion: v1
kind: Service
metadata:
name: headless-svc
labels:
app: headless-svc
spec:
ports:
- name: testweb
port: 80
selector:
ap^ ~ ) - R * ( kp: h: 9 CeadlessT v 8 & E j-pod
cluY U = L Y 0 s }sterIP:a r X 4 None
---
apiVersion: apps/vU 8 S E J ] k1
kind: StatefL o MulSet
metadata:
name: statefulset
spec:
serviceName: headless-svc
replicas: 6
selector:
matchLabels:
app: headless-pod
template:
metadatE V # Sa:
labels:
app: headless-pod
spec:
containers:
-V H c / F 7  name: testhttpd
image: 192.168.171.151:5000/zyz:v1
ports:
- co5 [ O L p G E 2ntainerPoF f = U drt: 80
volumeMow s P Runts:
- name: test
mountPath: /usr/local/apache2/htdocs
volumeClaimTemplates:
- metadata:
name: test
annotations:
volume.beta.kuberneter G ] W js.io/storage-class: test-sc
sp7 P / m 7 [ C Q ,ec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Mi
[root@docker-k8s01 ~]# kubectl apply -f statefulset.yaml
//查看Pv / N t W 8 } 0 uod运行状态
[root@docker-k8v E F Vs01 ~]# kubectl get pod -w
NAME                                     READY   STATUS    RESTf x O j ? 9 @ARTS   Ai h yGE
nfs-cl V 8 z O r o E Aient-provisioner-89699f486-qg7qw   1/1     Running   0          33m
statefulset-0                            1/1     Running   0          12m
statefulr a - . }set-1                            1/1     Running   0          42s
statefulset-2                            1/1     Running   0          36s
sa N B B 9 mtatefulset-3+ b  l } b P                            1/1     Running   0          33s
statefulset-4                            1/[ ^ @1     Runni{ ` i 1 Ong   0          30s
statefulset-5                            1/1     Running   0          26s
//查看pv及pvc的状态是否为bound,是否已经自动创建
[root@docker-k8s01 ~]# kf  : + ( v :ubectl get pv,pvc
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATU( % - k o / ` B 4S   CLAIM                        S5 D N B fTORAGECLASS   REASON   AGE
persistentvolume/pvc-050c1c1b-bd54-43ff-a654-1783003d350a   100Mi      RWO            Delete           Bound    default/test-statefulset-i } H g C # 1 z5   te_ l * T - ( W $st-sc                 2m2s
persistentvolume/pvc-53ba1821-5bed-4258-8291-604ca1656cda   100Mi      RWO            Delete           Bound    defaE j V J @ a m f tult/test-stz C $ -atefulset-2   test-sc                 2m13s
persistentvolume/pvc-b45b773a-475e-4e47-a670-436a604647d7   11 O _ - # u Z00Mi      RWO            Delete           Bound    default/test-statefulset-3   test-sc                 2m10s
persistentvol~ 0 l :ume/pvc-c9a44625-15d4-4ce9-8177-2e6c72178dd7   100Mi      RWO            Delete           Bound    default/test-k G statefulset-4   test-sc                 2m7s
persistentvolume/pvc-f0f4f4c1-6a00-498d-92a9-b794622dce3e   100Mi      RWO            Delete           Bound    def; { - m + +ault/test-statefulset-0   test-sc                 2m22s
persistentvolume/pvs G s @c-f7d3c7bu  ? F 1 62-c16d-4204-b2d7-1k t ! I % $4d3377b48d1   100Mi      RWO            Delete           Bound    default/test-statefulset-1   test-sc                 2m19s
NAME                                       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/test-statefulset-0   Bound    pvc-f0f4f4c1-6a00-498d-92a9-b794622dce3e   100Mi      RWO            test-sc        13m
perx l 5sistentvolumeclaim/test-statefulset-1   Bo5 q i * x ~ ] vund    pvc-f7d3c7b2-c16d-4204-b2d7-14d3377b48d1   100Mi      RWO            te_ r ` ^ b - ^st-sc        2m19s
persistentvolumeclaim/test-statefulset-2   Bound    pvc-53ba1821-5bed-4258-8291-604ca1656cda   100Mi      RWO            test-sc        2m13s
persistentvolw m F q E t M C numeclaim/test-statefulset-3   Bound    pvc-b45b7 3 2 L z ( e K73a-475e-4e47-a670-436a604647d7   100Mi      RWO            test-sc        2m10s
persistentvolumeclaim/test-statefulset-4   Bound    pvc-c9a44625-15dx } y % q 74-4ce9-8177-2e6c72178dd7   100Mi      RWO            test-sc        2m7s
persistentvolumeclaim/test-statefulset-5o ; _ _ ? x h - Q   Bound    pvc-050cU w D : ( ^1c1b-bd54-43ff-a654-1783003d350ae ` p   100Mi      RWO            test-sc        2m3s

2、完成之后,要求第0--5个PodA # C 8的主t p -目录应该为: Version:-# N m ; R B N-v1

将服务进行扩容:副本数量更新为10个,验证g % 9 I c Y L 0是否会/ Q ` ! O 2继续为新的Pod创建持久化的PV,PVCh I 7 i a

//编写脚本定义首页

[root@docker-k8s01 ~]# cat a.sh
#!/bin/bashe | D v &
for i in `ls /nfsdataW * e S _ b G`
do
echo "test---v1" > /nfsdata/${i}N V @ =/index.html
done
//查看节点IP,随机验证首页文件
[root@docker-k8s01 ~]# kubI # ! )ectl get pod, [ l X U Q -o wide
[root@docker-k8s01 ~]# curl 10.244.1.3
test---v1
[root@docker-k8s01l _ z | ~]# curl 10.244.2m g F [ J ` ; B.2
test---v1
[root@docker-k8s01 ~]# curl 10.244.2.3
test---v1
[root@docker-k8s01 ~]# curl 10.244.1.4
test---v1
//进行扩D ( I P % g 容更新
apiVersion: v1
kind:B D 4 # ] P Service
metadata:
name: headless-svc
labels:
app: headless-svc
spe0 q v v O c w Gc:
ports:
- name: testweb
port: 80
selector:
app: headless-pod
clusterIP: None
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: statefulset
spec:
updateG 9  | } 3 B G 4Strategy:
rollingUpdate:
partition: 4
serviceName: headless-svc
replicas: 10
selector:
matchLabels:
app: headless-pod
template:
metadata:
labels:
app: headless-pod
spec:
cg ) ^ e l & & Zontainers:
- name: testb B Uhttpd
image: 192.168.171.151:5000/zyz:v2
ports:
- containerPort: 80
volumeMop X M A %unts:
- na7 { u I . J ( 7 xme: test
mountPath: /usr/local! z + V g x J/apache2/htdocs
volumeClaimTemplates:
- metadata:
name: test^ O ` 5 C 3 O
annotations:
volume.beta.kubernetes.io/stora + | - = w @ge-class: test-sc
spe3 } Qc:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Mi
//查看其更新过程
[root@docker-k8s01 ~]# kubectl get pod -w
NAME                                     READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-89699f486-zqxsm   1/1     Running   0          18m
statefulset-0                            1/1     Running   0          17m
statefulset-1                            1/1     Running   0          17m
statefulset-2                            1/1     Running? W p H    0          17m
statefulset-3                            1/1     Run[ Q Nning   0          16m
statefulset-@ s 5 R P *4                            1/1     Running   0          16m
statefulset-5                            1/1     Running. m t I t w J 6 Z   0          16m
statefulset-6                            1/1     Running   0          7s
statefulset-7                            1/1     Running   0          4s
statefulset-8                            0/1     Pe# P 4 bnding   0          1s
statefulset-8p ( ( ( 8 | 8 2                            0/1     Pending   0          1s
statefulset-8                            0/1     ContainerCreating   0          1s
statefulset-8                            1/1     Running             0          3s
statefulset-9                            0/1     Pending             0          0s
statefu1 g ; g h *lset-9                            0/1     Pending             0          0s
statefulset-9                            0/1     P? @ Kending             0          2s
statefulset-9                            0/1     ContainerCreating   0          2s
statefulset-9                            1/1     Running             0          3s
statefulset-5                            1/1     Terminating         0          16m
statefulset-5                            0/1     Terminating         0          16m
statefulset-5                            0/1     Terminating         0          17m
statefulset-5                            0/1     TerminatinQ K Z T G # 9 wg         0          17m
statefulset-5                            0/1     Pending             0          0s
statefulset-5                            0/1     Pending             0          0s
statefulset-5                            0/1     ContainerCreating   0          0s
statefulset-5                            1/1     R~ 6 B a ; lunning             0          2s
statefulset-4                            1/1     Terminat` j G i $ - B R ring         0          17m
statefulss n 1 ` ^ Met-4                            0/1     Terminating         0          17m
statefulset{ a a s s 4 ( 7 A-4                            0/1     TermD B M a l T | Cinating         0          17m
sta% { wtefulset-4                            0/1     Terminating         0          17m
statefulset-4                            0/1     Pg _ p wending             0          0s
statefulset-4                            0/1     Pending             0          0s
statefulset-4                            0/1     ContainerCreating   0          0s
statefulset-4                            1/1     Running             0          0s
//查看其为扩容后的容器创F J v k ; ! r建的pv及pvc
[root@docke1 L !r-k8s01 ~]# kubectl get pv,pvc
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POM * _ B d { 5 { pLICY   STATUS   CLAIM/ 3 c | F                        STORAGECLASS   REASON   AGE
persistentv@ . _ _ 7 ;olume/pvc-2a4e3249-8513-4e42-P U d y :b883-aca51399d66b A m 47   100Mi      RWO            Delete           Bound    default/test-statefulset-3   tK j C R x u hest-sh T L J ( , 9 | Tc                 19m
persistentvolume/pvc-394568f7-5l | ^ 0 U ea38-413f-aaf2-4fa9f452a515   100Mi      RWO            Delete           Bound    def- : B ; n raulg w . ] F @ %t/testS J D ! @ d j 1-statefulset-8   test-sc                 2m25s
persistentvolume/pvc-40149679-e321-450d-8286-f4cd9a67f20f   100Mi      RWO            Delete           Bound    default/test-statefulset-5   test-sc                 19m
persistentv, ? : *olum7 [ u 9 Le/pvc-4f0a4fb C F 0 & e } / ;e3-8fa1-4bb9-a9 R nb7f-8652e2f4667c   100Mi      RWO            Delete           Bound    default/test-statefulset-0   test-sc                 19m
persiste: i { / Z _ntvm ] O 0 *  } m [olume/pvc-6c9bb0ed-4705-451b-a75e-2876c83a9cee   100Mi      RWO            Delete           Bound    default/e 2 q p 6 ,test-statefulset-7   test-Q $ d ^ /  Msc                 2m28s
persistentvolume/pvc-88652b42-f9cb-4c83-8660-5edfb2c7476f   100MiJ M 6 : 0 f x      RWO            Delete           Bound    default/test-statefulset-4   tek i { _ ; Rst-sc                 19m
persistentvolume/pvc-8d20cbb9-6aac-4ea6-8517-bd0c124b37a4   100Mi      RWO            Delete           Bound    default/test-statefulset-9   test-sc                 2m22s
persistentvolume/pv( l o ! , R 4 u ic-8d65da35-2219-459b-a26d-6e7ee0b7be48   100Mi      RWO            DeletZ y n f $e           Bound    default/test-statefulset-6   test-sc                 2m31s
persisteK J V U w S @ lntvolume/pvc-c2efd8ce-f90b-4ea ^ S / 6 Y X % 1ef-8f49-cdc55c375701   100Mi      RWO            Delete           Bound    default/test-statefulset4 Z W-2   test-sc                 19m
persistentvolume/pvc-e2a2b826-b8c3-485f-89ce-d0e54b3012f2   100Mi      RWO            Delete           Bound    default/test-statefulset-1   test-sc                 19m
NAME                                       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   S! ] s c ETORAGECLASS   AGE
persisten& L b Q } j N gtvolumeclaim/test-statefulset-0   Bound    pvc-4f0a4fe3-8fa1-4bb9-ab7f-86s ` 0 b52e2f4667c   100Mi      RWO            test-sc        1N F w { d h 3 x9m
persistentvolumeclaim/test-statefulset-1   Bound    pvc-e2a2b826-b8c36 j % V d (-485f-89ce-d0e54b3G 1 f 2012f2   100Mi      RWO            test-sc        19m* d , t $ i
persistentvolu2 u m 1 H jmeclaim/test-so F { 3tatefulset-2   Bound    pvc-c2efd8ce-f90b-4eef-8f49-cdc55c375701   100Mi      RWO            t0 T 0 8 f d $ West-sc        19m
persistentvolumeclaim/test-statefulset-3   Bound    pvc-2a4e3249-8513-4e42-b883-aca5p N 9 i Z1399d667   100Mi      RWO            test-J P w R ( _ dsc        19m
persistentvolumeclaim/test-statefulset-4U z  h W ! r #   Bound    pvc-88652b42-f9cb-4c8R N o : h )3-86e { b W = 6 B z60-5edfb2c7476f   100Mi      RWO            test-sc        19m
persisteJ ? 4 ? j q 7ntvolu4 o { M meclaim/test-statefulset-5   Bound    pvc-40149679-. v Ye321-450d-8286-f4cd% f &9a67f20f   100Mi      RWO            test-sc        19m
persistentvolumeclaim/test-statefulset-6   Bound    pvc-8d65da35-2219-459b-a26d-6e7ee0b7be48   100Mi      RWO            test-sc        2m31s
persistentvolumeclaim/tesW h M d 4 Zt-statefulset-7   Bound    pvc-6c9bb0ed-4705-451b-a75e-& u [ R w2876c83a9cee   100Mi      RWO            tesZ .  c pt-sc        2m287 r { F ys
persJ | T h S k k 9 yistentvolumeclaim/test-statefulset-8   Bound    pvc-394568f7-5a38-413f-aQ m zaf2-4fa9f452a515   100Mi      RWO            test-sc        2m25s
persistentvolumeclaim/te( S s 5 c ast-sta6 b m P xtefulset-9   Bound    pvc-8d20cbb9-6aac-4e# l R  4a6-8517-bd0c124b37a4   100Mi      RWO            test-sc        2m22s
[root@docker-k8s01 ~]# kubectl get pod -o wide
NAME                                     READY   STATUS    RESTARTS   AGE     I3 e 6 D k P LP           NODE           NOMINATED NODE   RE] Z 8 B B 6 EADINESS GATES
nfs-clie$ / 4 1 O  m @ Jnt-provisioner-89699f486-zqxsm   1/1     Running   0          21m     10.244.1.2   docker-k8s02   <none>           <none>
statefulset-0                            1/1     Running   0          20m     10.244.1.3   docker-k8s02   <none>           <none>
statefulset-1                            1/1     Run% E [ | ( A 2niK Q j o s P c Png   0          19m     10.244.2.2   docker-k8s03   <none>           <none>
statefulset-2                            1/1     Running   0          19m     10.244.2.3   docker-k8s03   <none>           <none>
statefulset-3                            1/1     Running   0          19m     10.244.1.4   docker-k8s02   &l1 k 0 c dt;none>           <none>
statefulset-4                            1/1     Running   0          2m15s   10.244.2.7   docker-k8s03   <none>           <none>
statefulset-5                            1/1[ @ G O x q F     Running   0          2m27s   10.244.1.8   dY b Gocker-k8s02   <none>           <d u 9  C 0 t 6 onone>
statefulset-6                            1/1     Running   0          2m52s   10.244.2.5   docker-k8s03   <none>           <none>
statefulset-7                            1/1     Running   0          2m49s   10.244.1.6   docker-k8s02   <none>           <none>
statefulset-8                            1/1     Re I ] Q nunning   0          2m46s   10.2Z p _ x 1 v . f c44.2.6   docker-k8s03   <none>           <none>
statefulset-9                            1/1     Running   0          2m43s   10.244.1.7   docker-k8s02   <none>           <none>
//访问查看其首页文件
[root@docker-k8s01 ~]# curl 10.244.2.5
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//~ m n j v QEN">
<html>
<head>
<title>Index of /</title>
</head>
<body>
<hn  f r1&gr p Q } *t;Index of /</h1>
<^ I ^ E g * z _ W;ul></ul>
</body>&~ E 0lt;/htmlY O 1 n>
[root@docker-k8s01 ~]# curl 10.244.1.8
test---v1

服务进行更新:在更新过程中,要求i0 S V i s u `d3以后8 8 * : g A的全部更新为Version:v2

//编写脚本文件
[root@docker-k8s01 ~]# cat b.A | X z M p v a hsh
#!/bin/bash
f* ? A s -or i in `ls /nfsdata/`
do
if [ `echo $7 K k % u *i | awk -F - '{print $4}'` -gt 3 ]
then
echo "test---v2" > /nfsdata/${i}/index.html
fi
done
[room # 6 . + V 1 0t@docker-k8s01 ~]# sh b.sh 

//查看Pod I| % U $ m V ` yP,访问验证是否- K P p更新成功


[root@docker-k8s01 ~]# kubectl get pod -o wide
NAME                                     READY   STATUm b *S    RESTARTS   AGE     IP# S k           NODE           NOMINATED NO5 c : ! k UDE   READINESS GATES
nfs-client-provisioner-8969O r k9f486-zqxsm   1/1     RuO H t N W + pnning   0          25m     1? 5 v M = d v , =0.244.1.2   dock4 - _ a w ? r fer-k8s/ B $ k L 02M h v Q [ Q   <n? | @ 6 | d H wone>           <none>
statefulset-0                            1/1     Running   0          24m     10.244.1.3   docker-k8s02   <none>           <none>
statefulset-1                            1/1     Ru* c y }nning   0          24m     10.24[ = ( D 7 9 D4.2.2   docker-k8s03   <none>           <none>
statefulset-2                            1/1     Running   0          24m     10.244.2.3   docker-k8s03   <none>           <nonex | g>
statefulset-3                            1/1     Running   0          24m     10.244.1.4   docker-k8s02   <none>           <none>
statefulsA D z _et-4                            1/1     Running   0          6m36s   10.244.2.7   docker-k8s03   <none>           <none>
statefuW b | $ @ alsetb f U 0 A-5S { S ] 2 { 7 .                            1/1     Runnin` = : Jg   0          6m48D k ) h vs   1W E .0.244.1.8   docker-k8o l 9s02   <none>           <no5 2 6 ? v N j Z ^ne>
statefulset-6                            1/1     Running   0          7m13s   10.244.2.5   docker-k8s03   <none>           <none>
statefulset-7                            1/1     Running   0          7m10s   10.244.1.6   docker-k8s02i F O & c s N (   <none>           <none>
statefulset-8                            1/1     Running   0          7m7s    10.244.2.6   docker-k8s03   <none>           <none>
statefulsd T = } Let-9                            1/1     Running   0          7m4s    10.244.1.7   docker-k8s02   <none>           <none>
//访问statefulset4
[root@docker-k8s01 ~]# curl 10.244.2.7
test-Y ) G i ` g--v2
//访问statefulset0
[root@docker0 Y !-k8s01 ~]# curM - { c - kl 10.244X  * q.1.3
tt ` P G 9 8 | pest---z s ev1