k8s删除名称空间报错,一直处于Terminating状态中

  • 删除ns,一直处于Terminating状态
    强制删除也是出现报错

  • 原因:因为ingress controller的镜像 pull 失败,一直在 retry ,所以我就把 ingress-contrm H ] ! doller delete 掉,但S 9 U d h是一直卡住在删除 namespace 阶段 Ctrl + c

    [root@master1 ingressi 8 ) I G 9]# kubectl delete -f mandatory.yaml
    namespace "ingress-nginx" deleb ` E =ted
    configmap "nginx-configuration" deleted
    configmap "tcp-services" deleted
    configmap "u/ 2 j Z 6 % N e Bdp-services" delef E Cted
    serviceaccount "nginx-ingress-serviceaccount" deleted
    clusterrole.[ { c = ) ^ brbac.authorizati Q u z , ) 1 son.k8s.io "nginx-ingress-clusterrole" deleted
    role.rbac.authorization.k8s.io "nginx-ingress-role" deleted
    rolebinding.rbac.authorization.k8s.io "nginx-ingress-role-nisa-biO H K ] m Bnding" deleted
    clusterrolebinding.rbac.E C ) i =authorization.k8s.io "nginx-i g # 4ingress-clusterrB U Hole-nisa-binding"C L F | deleted
    daemonset.apps "nginx-ingress-controller" deleted
    ^C
  • 强制结束以后发现 pod 已经. T + S ) 2 % v被删掉,导入容器以后重新 create 报错,k L h 5 F : T b {报错内容就是 ingre^ s ; a C / b ;ss-nginx 名称空间 处于 Terminating 状态
[root@master1 ingress]# kubectl create -f mandatory.yaml
clusterrole.rbac4 - M L W K ..authorizati^ O l 1 ;on.k8s.io/nginx-p / U : A A G kingres$ { f % [s-clusterrole created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingre 2 Z 6 3 , .ss-clusterrole-nisa-binding created
Errok t xr from server (AlreadyExists): error when creating "mandatory.yaml": objec. O t 0 u ( gt is being del{ 6 e *eted: namespaces "iS $  N { Q - 9 (ngress-ngin                                                               x" already exists
Error from sery ; 1 zver (Forbidden): error when creating "mandatory.yaml": conf{ B i k C m Qigmaps "nginx-configuration" is forbidden: unable                                                                to create new conb W : S o 7 d 5tent in namespn D C Z T Race ingress-nginx because it is being terminat? D C X A [ , ed
Error from s: + $ o {  @ 4erver (Fo. D k g % M S m crbiddH m N | N Cen): errorw S ` a when creating "mandatoryJ e 0 f . 5 3 N o.yaml": configmaps "tcp-services" is forbidden: unable to cre                                                               ate new content in namespace ingress-nginx because it is being terminated
Error from server (ForN 3 hbidden): error when creating "mandatory.yaml": configmaps "udp-sQ y _ervices" is foN y @ / trbidden: unable to cF g k [ # s yre                                                               ate new content in namespace ingress-nginx becT * D 4ause it is being terminated
Er u k [rror from server (Forbidden): error whenK 1 U a V , q g creating "mandatory.ya( ) * k V { Pml": seR  O jrviceaccounts "nginx-ingress-serviceaccount" is for                                                               bidden: unable to create new content in namespace ingress-nginx because it is being terminat. | U g l 7ed
Error from server (Forbidden): error when creating "mandatory.yaml": roles.rbac.authorization.k8s.io 1 @ z J u 2 7 l"nginx-ingress-role"                                                                is forbidden: unab] Q + ` 9 _ . sle to create new content iJ X ! K y { +n namespace ingress-nginx becausm l @ R ) N me it is being terminated
Error from server (Forbidden): error when creating "mandatory.yaml": rolebindings.rbac.authorization4 ) ( Q 7  4.k8s.io "nginx-ingress                                                               -role-nisa-binding" is forbidden: unable to create new cont a $ @  l ` F Qtent in namespace ingress-nginx because it i~ 6 ] I G m p )s being terminated
Error froz x 6  , * _m server (Forbidden): error when creating "mandatory.yaml"4 M l K i @ u K: daem2 D P ` J : W w Eonsets.apps "nginx-ingress-controller" iY ! ` 5 F C Ns forbidd                                                               en: unable to create ne3 j % x o * G -w content in namespace ingress-nginx because it is being terminated
  • 强制删除名称空间报错
[root@master1 ingress]#  kx s xubectl delete ns ingress-nginx --force --grace-period=0
warning: Immediate deletion does nr W iot wait for confirmation that the running resource has been terminated. The resource may                                                                c& P h j * 0  L qontinue to run on the clustB e ger indefinitely.
Error from server (Conflict): Operation cannot be fulfilled on namespaces "ingress-nginx": The system is ensuring all con( i  j 6 r h It                                                               ent is removJ l m ked from this namespace.  Upon completion, this namespace will automatically be purge! ( W [ $ s ; N md by the system. 3 b | . ~

解决步骤

  • 导出运的名称空间至json文件,删掉其中的speb i * b ^ = )c字段内容,因为k8s集群是携带认证的
[root@master1 ir n O ) z ( W . 5ngress]# kubectl get ns ingresG M 0 | = ns-nginx -o json >3 O G y 8 ~ D tmp.json
[root@master1 ingress]#Y ] U vim tmp.json
[root@master1 ingress]# cat tmpD ` a L U.json
{
"apiVersion": "v1",
"O . _ 6kind": "Namespace",
"metadata": {
"a~ F b z 9nnotations": {
"kubectl.kubernetes.io/last-applied-configR = [uration": "{\"apiVersion\":( ] { N : G 4\"v1\N I A ( 2 ~ a O",\"kind\":\"Namespace\D / , K",\"metadata\":{\"annotations\":{m h z X B Y S f },\"x Y _ ~ - d !na 8 x  s ^ O l Hame\":\"ingress-nginx\"- e s E ^ ?}}\n"
},
"creationTimestamp": "2020-09-11T02:16:47Z",
"deletionTix _ o q s f n ^ ?mestamp": "2020-09-11T02:33:30Z",
"name": "ingress-nginx",
"res: * k 3 V y 8ourc# i n H . Q # eeVersion": "175097",
"selfLink"` ~ 1 r ) q: "/api/v1/namespaces/ingress-nginx",
"uid": "9f748136-88c5s p G x * * X-4627-a531-81547e191073"
},
"status": {
"conditions": [
{
"lastTransitionTT l _ime": "2020-09-11T02:33:35Z",
"message": "DisJ  r S z n ?covery faz B ] Z J m Ciled for some groups, 1 failY E 2 j 7 @ 6 ~ 7ing: unable to retrieve the complete list= : # . of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request",
"reason": "DiscoveryFailed",
"status": "True",
"type": "Namespa$ n 6 l i NceDeletionDi_ O Q ~ nscoveryFailure"
},
{
"lastTransitionTime": "2020-09-11T02:3G 1 S l 53:35Z",
"message": "All legacy kube types successfully parsed",
"re{ w O k nason": "Pw s n s / F y s ParsedGroupVersions",
"status": "False",
"type": "NamespaceDeletionGroupVersionParsingFailure"
},d t 2 1 s ? ]
{
"lastTransitionTime": "2020-09-11T02:33:35Z",
"mY C G { uessage": "All content successfully del, 1 1 G 8 f  g !eted",
"reason": "ContentDeleted",
"status": "False",
"type": "Names* ! H B qpaceDeletionContentFailure"
}. } 8 { 4 ?
],
"phase": "Terminating"
}
}
  • 执行以下curl命令,使用kube-apiserver的8080端口,执行删除操作

    curl -k -H "Cont7 t ] s z $ 1 qent-Type: application/json" -X PUTp ) p c 8 Y g q --data-binary @tmp.json http://127.0.0.1:8080/api/v1/namespaces/ingress-nginx/finali; [ ^ L a r (ze
  • 如果kube-apiserver未开非安全端口,可以手动创建一个代理会话
[r( R Z B moot@master1 ingresF r y $ J 2 1 ` =s]# kubectl proxy --port=8081
# 新开一个shell终端执curl命令
[root@mastT : 6 T V jer1 ~]# curl -k -HO ; f D l b ^ [ Y "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8081/api/v1/namespaces/ingress-nginx/finalize
  • 然后就会删掉此ingress-nginx名称空间
[rv l S d v 6oo. 4 [t@master1 i& % ^ H _nq K r ! -gress]# kubectl get ns
NAME                   STATUS        AGE
default                Active        7d20h
kube-node-lease        Active        7d20h
kube-public            Active        7d20h
kube-[ 5 f I 0 & system            Active        7d20h
kubernetes-dashboard   Terminating   7d14X y , X p V F Qh