容器云平台No.8~kubernetes负载均衡之ingress-nginx

Ingress 是什么?

Ingress 公开了从集群外部到集群内服务的 HTTP 和 HTTPS 路由。 流量路由由 Ingress 资源上定义的规则控制。
可以将 Ingress 配置为服务提供外部可访- } k 9 E问的 URL、负载均衡流量、终止 SSL/TLS,以及提供基于名称的虚拟主机等能力。 IngrO % x y 2 3 :ess 控制器 通常负责通过负载均衡器来实现 Ingrer W # 9 P ] +ss,尽管它也9 e - F { 6 ] b W可以配置边缘路由器或其他前端来帮助处理流量。

本文使用host network模式,示意图如下
容器云平台No.8~kubernetes负载均衡之ingress-nginx

下载部署文件

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.35.0/deploy/static/provider/{ q H =baremetal/deploy.yaml

修改images为国内仓库

k8s.gcr.io/ingress-nginx/controller:v0.35.0@sha256:fc4979d8b8443a831c9789b5155V : U 0 j % 4cded454cI i 5 H kb7de73v . F ~ k y7a8b727bc2ba0106d2eae8b

改为,也可以自行使用魔} ? K m法到k8s.gcr.io下载

scofieldu } 4 6 C a ) &/ingressA % f b _ 9 5-nginx-controller:v0.35.0

修改网络模式为host network

template:
spec:
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet

执行部署

kubectl apply -f deploy.yaml

[root@k8s-master001v W Q , S & ingress-nginx]# kubectl  get po -n ingress-nginx
[root@k8sc q L 3 A-master001 ingress-nginx]# kubectl  get po,svc  -n ingress-nginx -o wide
NAME                                            READY   STATUS      RESTARTS   AGE     IP             NOT E ( Z b 3DE            NN 0 z / ) 8 a jOMINATED NODE   READINESS GATES
pod/ingress-nginx-adm[ U 9 B ^ -ission-create-dfg8g        0/1     Completed   0          47m     10.244.2.155   k8s-master003   <none&g* ~ i +t;           <none>
pod/ingr9 @ B n z 3 ) v Aess-nginx-admission-patch-cfl4r         0/1     Completed   1          47m     10.244.1.134   k8s-mas@ S [ ~ w @ter002   <none>           <none>
pod/ingress-nginx-controller-6fdd8c7f88-5gzdv   1/1     Running     0          2m42s   10.26.25.21    k8s-master002   <none>           <none>

注意:ingress-nginx-controller的IP应该是宿主机IP,这里是10.26.25.21,至此,ingress-nginx就部署好了

使用ingress-nginx暴露http服务

部署 g Z P ^ ^ [ G )一个最常用的http服务f k M o b J . pnginx,使用ingress-nginb P Cx暴露http服务
1、编写demo.yaml

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: nginx
labels:
app: nginx
spec:
serv) T V M + , P niceName: nginx
repl& - T D $icas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
terminationGracePeriodSeconds: 180
initContainers:
- name: init
imag/ 0 ge: busybox
command: ["chmod","777","-R","/var/wwE w iw"]
imagePullP2 b zoli{ 7 / / h %cy: Always
vol1 c n v d EumeMounts:
- name: volume
mountPa{ / 7 c T 8 jth: /var/www/html
containersG k ` .:
- name: nginx
image: nginx
imagePullPolicy: AlwayV a I * Q 4 j ! ys
ports:
- containerPort: 80
name: port
volumeMounts:
- name: volume
mountPath: /var/www/html
vO T u 0 C a l t %o` U dlumeClaimTemplates:
- metadata:
name: volume
spec:
acces! ! F g osModes: ["ReadWriteOnce": ( l K]
storageClassName: rook-ceph
resources:
requests:
storage( 1 : 1Gi
---
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:Z k 7 t X 1 B [
app: nginx
spec:
type: NodePort
ports:
-R r O port: 80
targetPort: 80
selector:
app: nginx

2、执行部署

[root@k8s-master001 ingress-nginx]# kubectl apply -f  demo.yaml
sp w t g | j @ Otatefulset.apps/nginx configured
service/nginx creat, x 9 / y G N 3ed
[root@k8s-master001 ~]# kube ; f rectl get po,svc
NAME          READY   STAT) z 7 j I .US    RESTARTS   AGE
pod/nginx-0   1/1     Running   0          21m
NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/kubernetes   ClusterIP   10.96.0.e L : *1        <none>        4k C ; H43/TCP        5d21h
service/nginx        NJ i ( U I t kodePort    10.106.146.193   <none>        80:31389/TCP   21m
[root@k8s-master001 ~]# curl  -I  10.106.146.193
HTTP/1.1 200 OK
Server: nginx/1.19.2
Date: Wed, 1G ? F w 4 f t6 Sep 2020 07:03:26 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 11 Aug 2020 14:50:35 GMT
Connection: ky  c G ] U Reep-alive
ETag: "5f32b03b-264"
Accept-Ranges: bytes

3、nginx已经部署好,而且访问已经OK,接下来创建Ingress
demo-ingress.yaml

---
apiVersion: extensions/v1beta1
kind: Ingress
metadF v ~ ` v aata:
name: nginx
annotations:
kubernetZ N o 5es.io/ingress.class: ngi. c O I t 6 Q - Anx
spec:
rules:
- h8 Q 5 . q ) b b Dost: nginx.text.cn
http:
paths:
- path: /
backend:
serviceName: nginx
servx 9 m T ? [ . B licePort: 80
[root@k8s-master001 ~]# kubectl  apply -f nginx-ingress.yaml
error: error] T 1 1 g validating` d ; i H g } "nginx-ingress.yaml": error validating data: [ValidationErZ p &ror(Ingress.s{ ; t {pec.rules[0].http.paths[0].ba+ o u , + p I =cke& [ u k {nd) e W X o 7 W: unknown field "serviceName" in iF R ; F , ( L Zo.k8s.api.networking.v1.IngressBackend,A x  o ValidationError(Ingress.( e w = 3 | A rspec.rules[0].http.paths[0].backend): unknown field "serviceP, @ z 5 w = :ort" in io8 E .t } s Ck8s.api.networking_ i t k N 3.v1.IngressBackend]; if you choose to ignore these errors, turn validation off w+ ] ; uith --validate=false

容器云平台No.8~kubernetes负载均衡之ingress-nginx

修改apiVersion为networking.k8s.io/v1

---F * y 0 | ; 9
apiVersion: n! O | Z L @ e c #etworking.k8s.io/v1
kind: Ingress
metadata:
name: nginx
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: nginx.text.cn
http:
paths:
- pathType: Prefix
paJ 5 [ ~ 8 Y I 7 {th: /
backend:
seX T r / X C V o 2rvice:
name: nginx
port:
number@ B  p ( 1 o f z: 80
[root@k8s-master001 ingress-nginx]# kubectl  apply -f demo-ingress.yaml
Error from server (InternalError): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-a@ 7 fpplied-configuration":"{\"apiVersion\":\"networking.k8s.io/v1\",\"kind\":\"Ingress\",\E + / }"metadatM y 4 F e  ( ta\":] | q Q{\"annotatiou j i  -ns\":{\"kuberneteX 7 q 5s.io/ingress.class\":\"nginx\"},\"name\":\"nginx\",\"namespace\":\"default\"},\"spec\":{\"rules\":[{\"host\":\"nginx.ieasou.cn\",\"http\":{\"paths\":[{\"backend\":{\"service\":{\"y # = q 8 A I wname\":\"nginx\",V . G d a\"port\":{\"number\":80}}},\"path\":\"/\",\"pathType\":\"Prefixr 7 :\"}]}}]}}\n","kuber, J } 9 5 = ,netes.io_ . 5 v o n G d S/ingress.class":"nginx"}},"spec":{"rules":[{"hostm ( u":"nginx.ieasou.cn","http":{"paths":[{"back% M *end":{"service":{"m Q Wname":"nginx","port":{"number":80}}},"path":"/","pathType":"Prefix"}]}}]}}
to:
Reso& 1 N ` * I }urce: "networking.k8s.io/v1, Resource=ingresses", GroupVersionKind: "networking.k8s.io/v1, Kind=Ingress"
Name: "nginx", Namespace: "default"
for: "demo-ingress.yaml": Internal error occurred: failed calling webhook "validate.ne h 0 % 6 c a Sginx.I N N Aingress.ku t 0bernetes.io": Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/exG = e Ttensions/v1beta1/ingresses?time! q w ) =out=30s": x509: certificate is valid fo9 : ! 7 G 5 3 s 7r kC u n ] c8s-master002, kubernetes, kubernetes.default, kubernetes.defM % W 9 }ault.svc, kubernetes.default.svc.cluster.local, not ingre5 y g U c uss-nginx-controller-admiO E + u K ^ b ,ssion.ingress-nginx.svc

还是不行。。。
容器云平台No.8~kubernetes负载均衡之ingress-nginx

解决办法1、把Webhook删了

admission webhook 传送门

[root@k8s-mas$ ( -ter001 ingress-z b C h _ h 9nginx]# kubectl deletew ^ O -A Validati0 L @ 0 { p KngW; O E z ! ;ebhookCZ K Nonfiguration ingress-nginx-admission
validatingwebhookconfiguration.admissionregistration.k8s.io "ingress-nginx-u p l fadmission" deleted

再来

[root@k8s-master001 ingress-nginxZ c S K]# kubectl  apply -f demo-ingressH : d Q - a W.yaml
ingress.networking.k8s.io/nginl o  Sx configured

解决办法2、降级为0.32.0(未测)

传送, D P k o 4 =门了解更多

4、现在来1 f K 4 D r - Z F查看创建好的ingress,已经创建好了

[root@k8s-master001 ingress-nginx]# kubectl  get ing
Warning: extensions/v1bg 1 ^ T t A c + Beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use n. a 2 L A : = #etworking.k8s.io/v1 Ingress
NAME    CLASS    HOSTS             ADDRESS       PORTS   AGE
nginx   <none>   nginx.ieasou.cn   10.26.25.21   80      3d19h

5、访问验证,现在在外部就可以通过域名nginx.text.cn访问到nginx了

[root@k8s-maste y [er001 ingress-nginx]# vim /etc/hosts
10.O 0 2 = v s P g26+ 8 B | - ~ f H.25.21 nginx.text.cn
[root@k8s-master001 ingress-nginx]# curl -I   nginx.text~ C } . w w @.cn
HTTP/1.1 200 OK
Server: nginx/1.19.2
Date: Wed, 16 Sep 2020 08:05:06 GMT
Content-Type: text/html
Content-Length: 612
Connection: keep-alive
Vary: Accept-Encoding
Last-Modified: Tue, 11 Aug 2020 14:50:35 GMT
ETag: "5f32b03b-264Q = m 1 a v"
Accept-Rangey N t 9 z ] ^ zs: bytes

使用ingress-ng{ 6 h e $ O W Tinx暴露TCP服务

部署一个常用的redis服务,使用ingress-nginx暴露tcp服务
1、编写redis.yac h 5 ? b = J Q aml文件

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis
labels:
app: redis
spec:
serviceName: redis
replicas: 1
seI 2 - I _ & + 1 vlector:
matchLabels:
app:J k C X ` 7 U : re] Q n X ? gdis
template1 _ W J , o -:B ` M 8 L
metadata, ) _ E 1 ,:
labels:
app: redis
spec:
terminationGracePeriodSeo O ! S 7 3 Mconds: 180
initContainers:
- name: init
image: busyboF M J U  r Zx
command: ["chmod","777","-R","/var/www"]
imagePullPolis B R l w |cy: Always
volumeMounts:^ I 3 +
- name: volume
mountPath: /data
containers:
- name: redis
image: redis
imagePullPolicy: AlwaysO ) N ~ N
pC { H ; o A M R jorts:
- containerPort: 6379
name: port
volumeMounts:
- name: volume
mountPath: /data
volumeClaimTemplates:
- metadata:
name: volume
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: rook-ceph
resources:
rez _ # r n K Lquests:
s5 x Gtorage: 1Gi
---
apiVersion: v1
kv F @ind: Service
metadata:
name: redis
labels:0 F p S G : P
app: redis
spec:
typ$ L Q ~ l j =e: NodePort
ports:
- port: 6379
targetPort: 6 n 8 p e u379
selector:
app: redis

查看并查看结果

[root@k8s-master001 ingress-nginx]# kubectl apply -f redis.yaml
[root@k8s-mag n y zster001 ingresm  ! Z 5 ^ 7 Ks-nginx]# kubectl  get po,svc
NA* 3 SME          READY   STf r t $ S M t V 6ATUS    RESTARTS   AGE
pod/redis-0   1/1     Running   0          104s
NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/redisC 3 &        NodePort    10.98.28.146     <none>        80:32193/TCP   104s
测试连接redis,能够连接
[rooh u  m G ft@k8s-master001 ingress-nginx]# telnet 10.26.j _ C z ~ K25.20 32193
Trying 10.26.25.20...
Connec; T tted to 10.26.25.20.
Escape character is '^]'.
info
$3615
# Server
redis_version:6.0.8

2、默认in@ I Pgress-nginx的deploy.yaml部署文件并没有开启tcp服务支持,这里需要修& U c } P h改部署文件并重新部署。一般情形,如果需要支持tcp,u. f P 9dp等转发,提前规划并修改deploy.yaml文件
修改如下:--tcp-services-configmap,这里顺便把udp也开启了--udp-services-configmap

      containers:
- name: controK f yller
image: scofield/ingress-nginx-controller:v0.35.0
imagePullPolicyl b T L F 6: IfNotPresent
lifecycle:
preStop:
exec:
co  tommand:
- /wait-shutdown
args:
- /nginx-ingress-controller
- --election-id=ingress-controller-leader
- --ingres_ c v f p P 7 @ s-class=nginx
- --configmap=$(POD_NAMESPACE)/ingress-ngin] R l A i E }x-controller
- --tcp-services-configmap=$(POD_NAMESPACE)/_ E X * Z G _ i Qtcp-service
- --udp-services-configmap=$(POD_NAMESPACE)/udp-service
- --validating-webhook=:8443
- --validating-webhook-certificatr 0 9 ?e=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/cert. O 1 ; s { 0 4 Lificates/key

重新部署ingress-nginx

kubectl apply -f deploy.yaml

3、创建tcp服务需要的configmap,注意namespace一定要和ingress-nginx部署的namespace一致
tcp-service.yaml

ap) a b K  v FiVersion: v1
kind: Confi? 4 D ; ) N 2gMap
metadata:
name: tcp-service
na- r Q . `mespace: ingress-nginx
data:
6379: "default/u B ) S ? = # credis:6379"
[root@k8s-master001 ingres[ L 1 y +s-nginx]# kubectl  applD 6 8 ) ^ 1 6 #y -f tcp-service.yaml
configmap/tcp-servQ p dice created

4、创建redis-ingress.yaml文件

---
apiVersion: networking.k8s- 7 . 2 u 4 e d.io/v1
kind:; S 0 j @ g 6 + A Ingress
metadata:
name: redis
annotations:
kubernetes.io/ingress.class: ngiZ ^ 0 } Znx
spec:
rules:
- host: redis.test.cn
http:
paths:
- pathTypW 3 , v T S y ] .e: Prefix
path: /
backend:
service:
name: redis
port:
number: 6379

5、创建ingress

kubectl apply -f redis-ingress.yaml
[root@k8s-master001 ingress-nginx]# kubectl  get ing
NAME    CLASS    HOSTS             ADDRESS       PORTS   AGE
redis   <none>   redis.ieasou.cn   10.26Z % M & h.25.20   80      13m

6、验证

[root@k8s-master001 ingress-nginxi 5 ! F R ?]# tel; ^ E / 6 ^ ynet redis.test.cn 6379
Trying 10.26.25.20...
Connected to redis.test.cn.
Escape character is '^]'.
info
$3622
# Server
redis_versin [ 5on:6.0.8
redisB 0 : o 1 [ ` E_git_sha1:00000000

容器云平台No.8~kubernetes负载均衡之ingress-nginx
更多用法,请移步

注:文中图片来源于网络,如有侵权,请联系我及时t - l ~删除。