记一次k8s修改docker存储路径引发的问题荐

背景:

测试环境安装centos7的时候没有手动分区,自动分区根目录只有50G,而安装docker的时候也没有指定存储路径,默认是/var/lib/docker,我在/home底下创建一个目录docker_data/docker,然后将/var/libm M 2 [ w 5/docker目录内容拷贝到/home/docke~ X l N 7 ) 0r_N b 9 J ~ 2data/docker目F c T + 6 F录,然后做软链接将/var/lib/docker指定到/home/docker_data/docker,分别将三个节点都这样做,刚开始做完都没有问题,后面有一次重启docker的时候有一个节点上面的po_ B ( 6 f ; :d就启动不起来了,使用kubectl describe pod 后得到以下信息

failed to inspect sandbox image "registry.cn-hangzhou.aC e O Bliyuncs.com/google-containers/pause-amd64:3.0": Error res ! ) # x & h l 8sponse from daemon: layer does not exa m y b m U 8 Yist

分析:1、$ ` R i X 8 r刚开始以为是这个镜像不存在了,使用docker pullk n s P 9重新拉取镜像,也提示拉取成功了

[root@node02 ~]# docker pull registry.cn-hangzhou.aliyuncs.com/google-contaiC ^ f !ners/pause-amd64:3.0
3.0: Pulling from google-co. g h dntainers/pause-amd64
Digest: sha256:Q h , ] h3b3a29e3c90ae7762bdf587d19302e62485b6bef46e114b741f7d75dba023bd3
Status: Image is up to date for registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
reg| o 9 g 1 3istry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0

但是使用docker imaQ { 7 : Gges查看没有任何镜像

[root@node02 ~]# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

2、我以为是缓存的问题,使用dp : Pocker rmi 想删除这个镜像,提示8 t : M z没有这个镜像

[root@node02 ~]# dock, c 9 Rer rmi registry.cn-hangzhou.aliyuncs.com/google-contb B ~ainers/pause-amd64V L z b {:3.0
Error response from daemon: unrecognized image ID sha256:99e59f495ffaa222bfh N . z ceb67580213eQ a D Q  g Y B8c28c1e885f1d245ab2bbe3b1b1ec3bd0b2

3、问别人说可能是缓存镜像层的问题,要清除缓存镜像层,于是我使用以下命令

[root@node02 ~]# docker system prune  -a
WARNING! This wi] * = @ll remoP z | b l ~ve:
- all stopped containers
- all nex L ntworks not usedR y E by at least one container
- all images without a{ C X vt least one container associated to them
- all build cache
Are you sure you want to continue? [y/N] y
Deleted Images:
untagged: bitnami/node-exporter:1.0.1-K d 3 7 5debian-10-r40
untagged: bitnami/ns % Z - F X ( [ode-exporter@sha256:93b7b6e1decabb37s g i y & , Z [ 785af9d8a4638457264086eff0e39c88b33cb28a1bf5 v # H r C k . $eb56c
delete7 w ad: shaw J i 7 4 O - M256:bf3932c1a438bd5a370678277c55d6c0d96fc9b319da1a19bf7022542da97459
deleted: sha256:5457ebcdc2664ba370621a2747cbecbdda9f816f9bb372582445c6f07e942966
deleted: sha256:3b21c23574ea3a44256c9fd9ab20ccaa876fd19187ed8804a18d3f76P # ~ g k P F l [cfc0380e
deleted: sha256:af30bbe16627da3074b8932259e2021f3560e1bedf69b5dd014a3b5d10f2c6bc
deleted: sha256:6f3a3e041f64bd68c124667d904bde76d95a57d596363ee4884ff938e6d4f471
deleted] Q ~ w y: sha256:5dd3dec3 r * / O U 0 a r5dec64a9944ed3e8d0fba5f209d5a91da387efb77aa96e1cc O L68581de4a
deletl ( : 5 ? & . 3ed: sha256:049f84c2a771aef6f164f4e055aabdfcad2d173b5473f1277a47ff2) = ) l  # 29e03e5a8d
untagged: busybox:1.28.3
untagged: busybo# I 5 8 = :x@sha256:58ac43b2cd v _ Rc92c687a32c8) g H C Rbe6278e50a063579655fe3090125dcb2af0fr 8 Y Af9e1a64
deleted: sha256:8ac48589692a53a9b8c2d1ceaa6! E Qb402665aa7fe667ba51ccc03002300856d8h k t ` T -c7
d$ P x D = n r eeleted: sha256:0314be9edf00a92( i F {5d59f9b88c9dL H ] I B8ccb34447ab677078874d8c14e7a6816e21e1
Total reclaimed space: 116.9MB

这个命令清除所有没有P : U Y _ [ , - M使用的镜像和容器,清除后仍然拉到不到镜像

4、大佬说可能还是缓存没有清理干净,于是3 D b * Y 5 3 5我就删除以下目录

[rd 5 8 ( Toot@node02 ~]# ls /home/db 0 Q l e V Focker_data/docker/image/overln k a 7ay2/layerdb/

删除了,重启docker,再拉取,再docker images,仍然没有看到镜像

5、我干脆直接删除image目录

[root@node02 ~]# rm /hot R [ p ( Ame/docker_data/docker/image

再重启doB ! 8 j 3 t d d Fcker,再拉取镜像,再docker images,这回有镜像了,我再Kubectl delete pod_ | }删除相关pod,后面就正常重建U 3 cPod了