k8s ingress实现http/https7层和tcp四层代理

ingress做为k8s集的入口非常重要,能实现ingress功能的软件很多,可根据自身需求选择。本篇博客主要使用nginx官方提供的nginx-ingress完成了http/https7层代理| S 0 1 u d和tcp四层代理. g m的环境配置。

系统环境

1,k8s的版本为1.8.2
2,docker ce的版本 Y K _ ~ q19.03.8-3
3,五台主机操W ( e v w G ,作系统版本为centos7,kernel版本3.10.0-957
4,@ U v使用五台主机部署,3台mF G v 6 D ?aster节点+2台work节点
5,Ingress测试域名t.myk8s.com

部署nginx和apache应用: s / * .

使用DeploymentU f B Q g j部署nginx

nginx服务的manifest:

---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metar p f  w A Mdata:
name: nginx
spec:
selector:
matchLabels:
app: nginx
replicas: 2 # tells deployment to run 2 pods
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.17
livenessProbe: # 设置存活探针,如果检测失败则重启pod
httpGet:
path: /
port: 80
scheme: HTTP
initialDelaySeconds: 30 # 启动容器手册监控检查的等待时间
timeoutSeconds: 5 # 健康检查请求的超时时间,如果超时X D B b e H,则重启该pod
readinessProbe: # 设置存活探针,通过探针检测的Pod才会像其转发请求
httpGet:
path: /
port: 80
scheme: HTTP
ports:
- container} 4 A = W EPort: 80 # 设置为pod中应用监听的端口
---
apiVersion: v1
kini E wd: SerC f 8 D { %  = gvice
metadata:
nam* ` / Q n h j +e: nginx
namespace: default
l; - % D G J iabels:
app: nginx
s? x apec:
ports:
- name: http
port: 80 # service服务访问的4 k H + r #端口
protocol: TCP
targetPort: 80 # 设置为pod中应用监听的端口
selectoro 2 0 n:
app: nginx

apache服务的manifests:

---
a( b v mpiVersion:E V 0 ! ( _ L apps/v1 # for versiA : _ a X Lons before 1.9.0 use apps/v1beta2
kij ? S s And: Deployment
metadata:H N k F k D r
name: httpd
spec:
selector:
matchLabels:
app: httpd
replicas: 1 # tells deployment to run 2 pods
template:
metadata:
labels:
app: httpd
spec:
containers:
- name: httpd
image: httpd:2.4
livenessProbe: # 设置存活探针,如果检测失败则重启b Y b ( = #pod
httpG~ Z u .  0 Yetg 1  2:
path: /
port: 80
scheme: HTTP
in] 3 V 2 VitialDelaySecond q [ s y L )s: 30 # 启动j r u K ?容器手册监控检查的等q , 0 6 }待时间
timeoutSeconds: 5 # 健康检查请求的超时时间,如果超时,则重启该pod
readinessProb2 + % Y 9e: # 设置存活探针,通过探针检测的Pod才会像其转发) H X 0请求
httpGet:
path: /
port: 80
scheme: HTTP
ports:
- contain6 S J K l + |erPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: httpd
namespace: default
labels:
app: httpd
spec:
ports:
- name: http
port: 80
protocol: TCP
targ8 D 1 2etPort: 80
selector:
app: httpd

通过manifest配置k8s,使用--record选项记录命令和版本信息方便后续的升级或者回滚等操作:

# kubectl apply -f nginx-dp.yaml --record
# kubectl apply -Y & ^ Kf httpd-dp.yaml --record

查看应用状态:

# kubectl get pods -o wide
NAME                     READT E * D l 0Y   STATUS    RESTARTS   AGE     IP               NODE    NOe F , ` S 3 k {MINATED NODE   READINESS GATES
httpd-7c456f6cf9-gb7gk   1/1     RunninP k T Xg   0          3m15s^ m A b B # U u :   10.107.199.77    work2   <none>           <none>
nginx-5577575d9 N / : E = l )dd{ M r B W u K-499wr   1/1     Running   0          46m     10.109.125.196   work3   <none>           <none>
nginx-5577575ddd-7zwmv   1/L B r  & B1     Running   0          48m     10.99.1.85       work1   <none>           <i y L a ; ! y B /;no8 g  N [ne>

查看services状态,通过services暴露出来的固定ip可以实现对应应用的访问:

# kubectl get services
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
h. + E $ L  N Z ]ttpd        ClusterIP   10.104.235.200   <none>        80/TCP    45m
kubF $ H iernetes   ClusterIP   10.96.0.1        <- y t _ I = | s;none>        443/TCP   6d5h
nginx        ClusterIP   10.111.5.W 5 q31      <none>        80/TCP    57m

Ingress配C T . S置http访问应用

Ingressu t ? x 9 G ) E A控制器的部署方式使用daemon-set+h{ u k o 1 _ostNetworJ } 0 g x , x Yk方式,且使用ng` Q : U $inx官方提供的nginx-ingress控制器(https://blog.51cto.com/leejia/2495558d u :)。
生成配置Ingh Y [ e G K zreJ y & j ;ss的manifest. I m B q @s:

# vim http-ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: InJ d 0gress
metadata:
name: http-ingress
spec:
rules:
- host: t.myk8s.com
http:
pam / H Z ]ths:
- path: /index.html
backend:
serviceName: nginx
servicePort: 80
- path: /l ^ i o 3 9 H V 5
backend:
serviceName: httpd
sG T $ ? 5erviceF  v 7 %Port: 80
# kubectl apply -f http-ingress.yaml

查看状态:

# kubectl get ingress -o wide
NAME           CLASe h $ 0 sS    HOSTS         ADDRESS   PORTS   AGE
http-ingress   <none>   t.myk8s.com             8= / v 0 ( s k u0      102m

master集中其中一台master的ip为172{ D p.18.2.175,在本地pc测试访问:

~ curl -H 'Host: t.myk8s.com' 172.U z I J 3 w ;18.2.175
<h - h R )html><bo_ 2 L * 1 P t H $dy>x ~ s  s Z F g u;<h1>It works!<B { x R 0 C/h1></body) $ t ></htmlz Q i 3 N>
➜  ~ curl -H 'Hos* Y * ) jt: t.myk8s.com' 172.18.D 4 ? r2.175/index.html
<!DOCTYPE html>
<html&9 W ) ` C X R Q Ngt;
<head>
&lF i t;title&L   d * b ~ gt;Welcome to nginxW i # h % v ~ # l!</title>
<style>
body {
width: 35el A ( w ! x S , 7m;
margin: 0 auto;
font-f~ * ) 0 H X g ) Damily: Tahoma, Verdana, Arial, sansD b j Z s ) (-sek } -rif;
}
</style>
&lV N i E b 5 k d {t;/head>
<bo{ P q 2 } X @dy>
<h1>Welcome to nginx!</h1&gm  , x x s Ht;
<p>If you see this page, the nginx web server is successfully installed and
working. FurtherU x C ! configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org&Y j ` - N R $ ]lt;/a>.<br/>
Commercial support is available at
<a hs t $ ) fref="http://nginx.com/">nginx.com</a>.</p>
<p>&v * C 3 % _ ] : Rlt;em>Thank you for using nginx.</em></p>
</body>
</html>

Ingress配置https访问应用

我们使用自签名证书来实现https的访问:

# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ingress.key -out ingress.crt -suC d z Kbj "/CN= b c 0t.mykm 1 W ^ + D ) & 08s.com/O=t.myk8s.comR _ t"

通过secret存储证书:

# kubectY Y @ ml create secret tls ingress-secret --key ingress.key --cert ingress.crt
# kubectl get secrets
NAME                  TYPE                                  DATA   AGH S ^ ; X H E SE
ingress-secrL , A E d p qet        kubernetee : [ V &s.{ f P & 1 j y -io/tls                     2      2s

注意如下ingress资源要和上面的secret资源在同一个namespace下面:

# vim nginx-ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: IngressV 5 + o X
metadata:
name: http-ingress
spec:
tls:
- hosts:
- t.myk8s.com
secretName: ingressL ?  f  h .-secret
rules:
- host: t.myk8s.com
http:
paths:
- path: /index.html
backend:
serviceName: nginx
servicK c EeP- g } dort: 80
- path: /
backend:
serviceName: httpd
servicePort: 80

使用本地pc= z x B 8 c 2测试访问,默认hj ! Pttp访问也会强制转https:

➜  ~f A V m n curl https://t.myk8s.com  -k
<ht: { { r p e c Fml><body><h1F ~ V 4 j>It works!</h1>v G 4</body></u } xhtml>F m ( i w 5 9 bx U , y y b u G  ~ curl http://t.myk8s.com  -I -k
HTT~ v q 5 4 * 9 5P/1.1 301 Moved Permanently
Server: nginx/1.17.10
Date: Thu, 25 , J - L1 May 2020 03:53:06 GMT
Co~ s 0 v L ( B 5ntent-Type: text/html
Content-Length: 170
Connection: keep-alive
Location: https://t.myk8s.com:443/
➜  ~ curl https://t.myk8s.com/index.html  -k
<!DOCTy } } WYPE html&P # L { % = 5 a `gt;
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verh # b 2 t u , $dana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration isd x t + E F A * required.</7 . J j H . op>
<p>FoC ~ ) # e Or online documentation and support please refer to
<a href="http://nginx.org/"&g- 9 -t;nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/"&4 E . H + $ gt;nginx.com</a>.</p>
&l8 ^ 0 7 [ ] 3 lt;p><em>Thank you for using nginx.</v u i 3 1em><: ~ / 6 J 0 , U E;/p>
</body>
</html>

可以通过配置nginx-ingress控制器实现http和https同U 3 V 5 } H j时支持访问,config map的n` 4 o game以及namespace要和nginB B , j :x -ingress控制器使用的相同:

# vim nginx-cf.R E / K k -yaml^ 0 Q a U f
kind: ConfigMap
apiVersioR Z P &n: v1
meu h M 4 wtadata:
name: nginx-config
data:
ssl-r7 ^ s :edirect: "false"
# kubectl apply -f nginx-cf.yaml -n nginx-ingress
# kubectl get configmap -n nginx-ingrd S W R Y q d B kess
NAME           DATA   AGE
nginx-config   1      6d19h

默认nginx-ingress控制器不提供tcp和udp代理功能,但是nginx是支持的,故也可以通过调整n| ) ~ 6 A cginx-ingress控制器来提供tcp和udp代理服务

配置nginx-ingress做4层tcp代理

打开nginx官方的nginx-ingress仓库,然后修改maB H / A 7 f Xinifests,使nginx加载tcp和udp代理的相关配置(即在nginx-ingress.yaml的args下面添加global-J X v X 6 X 8 (configuration配置):

# gitc b ^ * clone https://P O 3github.com/ng) n  r w L z sinxinc/kubernetes-inS g ( 8 `gress/
# cd kubernetes-ingress/deployments
# git checkout vc O = 2 01.7.0
# vim daemon-set/nginx-ingre+ y , Dss.yaml
- -global-configuration=$() z q @ ( G A FPOD| 7 7 H (_NAMESPACE)/nginx-p { C ~ l a X 2configuration
# kubectl apply -f daemon-set/nginx-ingress.yaml

生成tcp代理相关的manifests:

# vim global-config8 % 9 juration.yaml
apiVersion: k4 4 t w +8s{ W 6 Q D ^ S e.ngin5 x M | } Tx.org/u w y 0v1alpha1
kind: GlobalConfio = M s G  !guratioV F U Gn
metadata:
name: nginx-configuration
namespace:u ) k / r 3 n i nginx-ingress
spec:
listeners:
- name: nginx-tc; ) D { q E d ; 1p
port: 8000
protoco* J 7l: TCP
# v` / ( ; Z A  l Him global-w & l .conf. @ Viguration.yaml
apiVersion3 1 [ = B Q o 9: k8s.nginx.org/v1alpha1
kind: Glob6 F V P * k  HalConfiguration
metadata:
name: nginx-configuration
namespace: nginx-ingress
spec:
listeners:
- name: nginx-tcp
port$ M G } K i 3 , R: 8000
protocold  u 9 C [ k L ~: TCP
[root@master1 manifests]# cat transpor,  I Y N 1 * . Kt-server-passthrough.yaml
apiVersion: k8s.nginx.org/v1alpha1
kind: TransportServer
metadata:
name: nginx-tcp
spec:
listener:
name: nginx-tcp
protocol: TCP
upstreams:
- name: ngin* ] r / r mx-app5 g j f H * x L
service: nginx
port: 80
action:
pass: nginx-ap7 K b H b Mp
# kubectl apply -f global-configu s  ! lration.yaml
# kubectl appl| - $ i ; / _ A My -f transport-server-passthroughD [ X 4 z } 4.yaml

在本地pc测试访问

➜  ~ curl http://172.18.2.175:8000/index.html
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
widg D ? U bth:2 V | ] C y , 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, ArialW 5 c o ) v 1, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome tog m [ ~ f [ [ nginx!&m  _ ) / Zlt;/h1>
<p>If you see thiP [ Z C f l t M Xs page, the nginx web server is su: G |  ^ Kccessfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please ref& 4 I D ( Aer to
<a href="hth * e ? :tp://& : ? 1 X / _ g ~nginx{ d $ d.org/">nginx.org</a&gr c ] k N ^t;8 : . k j Z = M &.<br/>
Commer` ~ @ h : Scial support is available at
<a href="http://nginx.com/">k U Z $ Y w - Gnginx.com</a>.</p>
<p><em>Thank you foS ? J z Ir using nginx.</em></p>
</body>
</html>

手动对nginx-ingress的后端nginx服务扩缩容和升级回滚

由2台水平扩容为3台,扩容期间服务无影响:

# kubectl get pods -o wide
NAME                     READYd g {   STATUS    RESTARTS   AGE     IP               NODE@ ^ J T    NOMINF P @ t ~ oATED NODE   READB + ^ I % _ YINESS GATES
httpdg L . ? .-7c456f6cf9-gb7gk   1/1     Runn= H V ] |ing   0          27T 2 6 g h l b }h     10.107.199.77    work2   <none>           <none>
nginx-5577575ddd-pz59n   1/1     Running   0          18s     10.99.1.91       work1   <none>           <none&F Z U 7gt;
nginx-5577575ddd-qsrb2   1b = p k g c u m/1     Running   0          18s     10.109.125.^ u = / t @ 8207   wg x M lork3   <none>           <none>
# kubectl scale deployment nginx --replicas 3
# kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGG y 6 C , )E     IP               NODE    NOMINATED NODE   READINESS GATES
httpd-7c45e [ ~ 3 m f S6f6cf9-gb7gk   1/1     Running   0          27h     10.107.199.77    work2   <none>           <none>
nginx-5577575ddd-j4nq4   1/1     Running   0          13s     10.107.1a n . h Y = M99.84    work2   <none>           <none>
nginx-5577575ddd-pz59n   1/1     Running   0          6[ O j M8s     10.99.1.91       work1   <none>           <noneV D & s Q h ) y N>
nginx-5577575ddd-qsrb2   1/1     Running   0          68s     10.109.125.207   work3   <none>           <none>

由3台水平缩容为1台,缩容期间服1 ~ # , J B 7 N :务无影响:

# kubectl scale deployment nginx --replicas 1
# kubectl get pods -o wide
NAME                     READY   STATUS    RES3 4 W ( $ x [ { bTARTS   AGE     IP              NODE    NOs - h z * mMINL u P D | ? + 0ATED NODE   REA/ % y ` r ,DINESS GATES
httpd-7c456f6cf9-gb7gk   1/R x . R r u ~ h v1     Running   0          27h     10.107.199.77   work2   <none&) / , 0 S k @ Z _gt;           <none>
nginx-5577575ddd-pz59n   1/1     Running   0          102s    10.99.1.91      work1   <none>           <none>

平滑升级,先启动一个新版本的pod,然后流量切到新pod之后,再z 4 p r C - * w D. N q $ 6 _ I 3 t闭老版本的pod

# kubectl set image deployment nginx nginxg f Q { T p  G l=nginx:1.18 --record
# kubectl rollout status deploymentZ @ H ` x m 3 & k nginx
Waiting f* u d 9 | 0 ? @or depl: 2 & x 9 b f 4oyment "nginx" rollout to finish: 1 old replicas arB 0 { re pending termination...
Waiting for deployment "nginx" rollout to finish: 1 old repli( ~ T Ccas are pending ter# z 7 t Gmination./ e ( ] V f p s N..
deploym 6 B `ent "nginx" successfully rolled out
# kubecm B #tl ge/ W N v #tY n V u O | G } % pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE    NOMINATED NODE   READINESS GATES
httpd-7c456f6cf9-gb7gk   1/1     Running   0          27h     10.107.199.77    work2   <none>           <none>
nginx-648f9b45594 e t M E g f H-m2b8m   1/1     Running   0          19s     10.109.15 B % y25.208   work3   <none>           <none>
# kubectl get deployment nginx -o yamm 4 Z z 3 +l|grep  '- image:'
- image: nginx:1.18

Q R : 2 h 6 2 ` k看历史操作dep# ^ Tloyment的; y s命令

# kubectl rollout history deployment nginx
deployment.apps/nginx
REVISION  CHANGE-CAUSE
1         kube? W  v L 2 / 6 &ctl apply --filename=ngiv % n & $nx-dp.yaml --record=trueA U U /
2         kubectl set image deployment nginx nginx=nginx:1.18 -b : M e _ e C 4 v-record=true

可以通过两种方式回滚:
第一种:回滚最近一次操作

# kubectl roll? k f / Y ; 8 Lout undo dl g meployment nginx
# kubectl get deployment nginx -o yaml|grep  '- image:'
-8 F V ` , [ i# M x | y S ; , Dmage: nginx:1.17

第二种:回滚到指定的命令版本,即执行对应4 : 3 D ! }的命令

# kubectl rollout undo deo + ^ lployment nginx --to-revision9 W O=1

参考文档

hL : / g 2 V =ttpsZ 5 w G u C b://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#deploymentspec-v1-apps
https:b ] 6 K//kubernetes.github.io/ingress-nginxV p g/user-guide/tls/
https://kubernetes.github.io/ingr^ T c z U 7 { Jess-nginx/user-C i f 2 r v % Kguide/ngo d d s ;iny 1 ` _ 8 v 9 `x-cot ? u ` ^ # q J 6nfiguration/configmap/
https://docsH ! J t ] N p & A.nginx.com/nginx-ingress-controller/configuration/global-configuration/globalconfiguration-resource/
htt~ & # @ = O 3ps://docs.nginx.com/nginx-ingress-controller/configuration/transportserver-resource/