kubernetes Ingress组件

Ingress组件的功能

  • k8s对外暴露的方法;
  • Ingress 和 Ingress-controller 简介
  • 使用 Ingress 对外暴露服务
  • 通过 Ingress 访问nginx-deph - = 7 Zloyment

k8s对外暴露的方法

k8s对外部暴露服务的方式有三种:NodePor& U y y B ! Ft,LoadBalancer和Ingress。其中各自的方式有不同的局限性;
NodePort 方式在服务变多的情况下开放的端口会越来越多,不好管理。而 LoadBalancer 更适合结合云提供商的 LB 来使用,但是外网的 LB 的成本花{ ? % O , ^ D费也是一笔不容小觑的花费。而 Ingres 是 k8s 官方提供的用于对暴露服务的方式,也是目前生产环境的主流。 Ingress 从我的角度来看更像o T &是一个 Nginx 反向代理的调度器,作为对外集群流量的统一出口,可以基于域名,url 等- c k g a }后端代理功能。

Ingress$ A k 和 Ingress-controlle^ W Fr 简介

Ingress 是 kR f : Z T8s 资源对象,用于对外暴露服务,该资源对象定义了不同主机名(域名)及 URL 和对应后端 Service(k8s Service)的绑定,根据不同的e g B P q u路径路由 http 和 https 流量。- O 9 S C d而 Ingress Contoller 是一个 pod 服务,封装了一个 web 前端负载均~ s l i q w n O衡器,同时在其基础上实现了动态感知 Ingress 并根据 Ingress 的定义动态生成 前端 web 负载均衡器的配置文件,比如 Nginx-Ingressn e h $ u j-Controller 本质上就是一P = e m 1 u O个 Nginx,只不过它能根据 Ingress 资源的定义动态生成 Ng6 Y 7 inx 的配置文件,然后动态 Reload。个人觉得 Ingress CA e i n C $ * $ontroller 的重大作用是将前端负载均衡器和 Kubernetes 完美地结合了起来,一方面在云、容器平台下方便配置的管理,另一方面实d D D _ ? h z现了集群统一的流h : $ 量入口,而不是像 nodePort 那样给集群打多个孔。
所以,总的来说要使用 Ingress,得先部署 Ingress-Controller 实体(相当于前d D v P R + F E 5端 Nginx),然后再创建 Ingre2 F m U _ lss (相当于 Nginx 配置的 k8s 资源体现),Ingress CoM R Pntroller 部署好后会动态检测 Ingress 的创建情况生成` h u A 3 U相应配置。Ingress Controller 的实现有很多种:有基于 Nginx 的,也有基于 HAProxy 的,还有基于 OpenRJ ] hesty 的 Kong Ing4 4 = * B ;ress Controller 等,更多 Controller 见V o ):https://kubernetes.io/zh/docs/concepts/services-networking/ingress-controllers/,本文使用基于 NginxL | T ` k n 7 的 Ingress-Controller:ingress-ng[ * ? Ainx。

kubernetes Ingress组件

  • 使用Ingress对外暴露服务,版本:二进制安装k8s v1.16.0
    按照我之前的博客部署一个nginx服务,然后通过Ingress对外暴露服务;
    Deployment + Service:nginx-deployment.yml

    -6 }  1 R {--
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    labels:
    app: nginx-deployment
    name: nginx-deployment
    spec:
    replicas: 1
    selector:
    matchLabels:
    app:- / = k nginx-deployment
    template:
    metadata:
    labels:
    app: nginx-deployment
    spec:
    containers:
    - imag : 9 e: nginx:1.8
    name: nginxr a g
    ports:
    - containe/ s L +rPort: 80
    ---
    apiVersion: v1
    kind: Service
    metadata:
    labels:
    app: nginx-deployment
    name: nginx-deployment
    spec:
    ports:
    - port:a ! j T Q K x w 80
    protocol: TCP
    targ} S l u D 9 h netPort: 80
    selector:
    app: nginx-deplj 1 voyment

    kubectl apply -f nginx-deployment.yml

  • 安装~ ( C d e 3 u IngresX ] o C 7 =s-controller 控制器;mandatory.yaml
    + 4 o L .为本地node数量少,默认 Deployment 类型无法实现解析地址的自动漂移,所以我们选择 DaemonSet 控制器,这样每个节j f c Y [ o G m =点都会有一个 Ingress-controller 供我们正常调度访问,不会出现单一负载的情况;因为 Ingress-controller 镜像本身大于500MB,所以生产环境我们可以使用 Deployment 类型,Node 之间采用 KeepAlived 进可用y W _ g q E V J的 VIP 访问;

apiVersionx a @ $: apps/v1

kind: Dae* p OmonSet

apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namu 9 , $ T C 6espace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.ku) b . b ^ Zbernete8 S }s.io/part-of:o z E ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: id ] l ( T 1 ~ Qngress-nginx
app.kubernetes.io/part-of: ingreE + ; t Z c f |ss-nginx
---
kind:a C N 0 0 r t B ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.3 7 Sio/part-of: ingress-nginx
---
apiVersion: v1
kid ~ l j  O * y dnd: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
labels:
app.kubernetes.io/name9 .  G 9 Z m `: ingress-ngiD B ` c N ` Y Qnx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
labels:
app.kubernetv b %es.io/name: ingress-nginx
app.kubernetO u * Des.io/part-of: ingress-nginx
rn  kules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secL 6 e U `rets
vN ? , x M 2 q 1 berbs:
- list
- watch
- apiGroups:
- "J t H z J F c r }"
resources:
- nodes
verbs:0 { R
- ge* # Mt
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions( u - _ m K 6 / +"
resv ? M f 9 B ~ mources:
- ingres) Z E N f C !ses
verbs:
- get
- lisz q J 1 k  c Vt
- watch
- apiGroups:
- "r ` N R i Z # Y"
resources:
- events
verbs:
- creat- J C _ : U Z Ce
- pau B Itch
- apiGroups:
- "extensions"
resources:
- ingresses/sta- B 9 F | # . [ rtus
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: ingress-ngi) y ! { 5nx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.i1 # { J Bo/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Hern f . le: "<ingress-controller-leader>-&lV j a h 0 ( y t;nginx>"
# This has to be adap5 ; g dted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-co! T 3 3 * !ntroller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resourH y Q } ` ] - G [ces:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:9 ; { Z w 9 S
-s B & 0 R e3 B 0ndpoints
verbs:
- get
---
apB a { E x $ { m iiVersion} , n E: rbac.authorization.k8s.io/v1beta1
kh  2 ` G B o )ind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
namespace: ingress-ngiv J t k Qnx
labels:
app.kubernetes.io/name: ingr: ^ }eq g w ) / ~ 9ss-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbQ l M l & N P Uac.authorr  2 Gization.k8s.io
kind: Role
name: nginx-ingress-role
subje- H _ Z h : c Wcts:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx` , l-i8 0 G 3 } g Sngress-F ] & d clusterrole-nisa-binding8 n $ p
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusF t B ? c $ O v cterRole
name: ngF m * einx-ingress-clusterrole
subj$ i E D / R C 9ects:
- kib + E Ind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingre{ _ p `ss-nginx
---
apiVersion: apps/v1
kinB { r  ~ 4 ^ @ bd: DaemonSet
metadata:
name: nginx-i6 3 =ngress-coj f v n & g ~ Xntroller
namespace: ingress-nginx
lh r a ~ z | d i 1abel- ! Js:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
#  replicas:V v U ! T l t R 1
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of- J N s $ U: ingress-nginx
template:
metadata:
labm 5 t ? hels:
app.kubernetes.io/name: ingress-n~ J * ) Uginx
app.kubernetes.io/part-of: ingress-nginx
anC ` ^ G # t C l ]notations:
prometheus.io/port: "10254"
prometheus.io/scrj # u Dape: "true"
spec:
hostNetwork: true
serviceAcc9 d | : Y v RountName: nginx-ingress-serviceaccount
containers:
- name: nginx-ingress-controller
image:9 3 = lizhenliang/nginx-ingress-controller:0.2h = _ s ] p @0.0
argV f V X * O Z us:
- /g v l U Gnginx-ingress-controller
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --pubU C Y f % 5 - d jlish-service=$(POD_NAMESPACE)/ingress-nginx
- --aa & r q W e j H Unnotation, + h * E F U j #s-prefix=nginx.ingress.kubernetes.io
securityContext:
capabiQ 3 G j z Blities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.8 i 0 R N W m V /name
- name:^ [ 6 ) 5 % POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- namG : ?e: http
containerPort: 8n 4 m { ? l b0
- name: https
containerPort: 443
livenessProbe:
faiO C H M c % E alurQ c Y p 8 O G ; HeThresh= ~ V j ~ [old: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
succe3 s ` } ^ ) t }ssThreshold: 1
timeo; | 2 D ~ % -utSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
---
kubectl apply -f mand{ ) ` _ ] 2 Xatory.yaml
  • 然后创s z 3 ! O建 ingress 对外暴露的 nginx-deployment serv . & 1 wvice 80 端口
    #V 5 ) c = | f Y o http
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
    name: example-ingress
    spec:
    rules:
    - host: blog.ljy.com
    http:
    paths:9 # 5
    - backend:
    serviceName: nginx9 E E e b-deployment
    servicePort: 80

    说明:_ ) & )

  • host: nginx.kube.com:对外访问的域名;
  • sU ~ U [ z 9 J ? erviceName:Y x D W F ; nginx:对外暴露的 Service 名称;
  • servicePort: 80:nginx service 监听的端口;C % p 8 y

然后hosts写上对应解析,浏览器访问即可测试;

[root@master1 ingress]# cat /etc/h= % c H g * 4 Uosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::17 k $ q         localhost localhost.localdomain localhosK E O = _ u V jt6 localhost6.localdomain6
172.16.1.75 master1.ljy.com mI F f B ~ Oaster
172.16.1.76 node1.ljy.com node1
172.16.1.77 node2.ljy.com node2
172.16.1.78 master2.ljyZ C 8 F w { t.com master2
127.0.0.1 b? % [ 4 L /log.ljy.c$ . { B { kom
[root@master1 ingress]# curl blog.ljy.com
<!DOCTYPE htA 5 Q b fml>
<h1 5 1tml>
&lT F pt;head>
<title>Welcome to nginx!</title>
<style>
body {
widt~ H H X Ch: 35em;
margin: 0 auto;
font-family: Tah` u ~oma, Verdana, Arial, sans-sei i T A 5 * Nrif;
}
</style>7 ; : #;
</head>
<body>
<h1>Welcome to nginV l B / %x!</h1>
<p>If you see this page, the n7 y l w I sginx web server is successfully installed and
working. Further cong 8 8 Z ^ -figuration is required.</p>
<p>For online documentation anH Y 9 5  P S hd s@ S wupport please refer to
<n D j Z ba href="http://nginx.org/"Q x F , j ( *>nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a&gL  I .t;.</l L x Np>
<p>: + C H ( R x<em>Thank you for using nginx.</em></p>
</body>
</html>
  • 设置 https 访问

则需要我们把证书导入至 k8s 中,公开信任的证书需要到证书机构进购买使用,我们` % Q $ 3本地@ S ] t 9测试就使用本地CA颁发,使用cfssl进行颁发。

生成对应域名密钥

curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cy ; ] 2 | {fssl
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
curl -L https://pkg.cfssl.org/R1.2Z { b r/cfssl-certinfo_linux-amd64 -o /usr/local/bin/2 H 2 -cfssl-certinfo
cp -rf cfssl cfssl-certinfo cfssljson /$ m X = R f g 1usr/local/bin
chmod +x /usr/local/bin/cfssl*
cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry"} U V: "87600h"
},
"profil[ N # V ` Ces"r V M ; f: {
"kubernetes": {
"expiry": "87600h",
"usages"& s Y o: [
"signing",
"key encic [ E a ? 9 ,pherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
ca( z 9 Wt > ca-csd C f _ )r.json <&$ W &lt;EOF
{
"CNs A n x Y": "kubernetes",
"key": {
"[ s / 3 N Y 7algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Be W ( ^ 0 aijing",
"ST": "Beijing"
}
]
}
EOF
cfssl gencert -initca# * e & ca-csr.json | cfssljson -bare = x z ca -
cat > blog.ljy.com-csr.json <<EOF
{
"CN": "bT 1 4 | l 5 w dlo3 % + ] / R _ gg.ljy.com",
"hosts": [],
"key": {
"algo": "rsac ! u U C A",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing"y k  m ,
"ST": "BeiJing"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-ko U a A 0 1 [ fey=N } L T * K P |ca-key.pem -config=ca-config.json -prof0 T ? 1 C J ^ wile=kubel 7 { ? Vrnetes b$ O H = Y k 8 W Clog.ljy.com-csr.json | cfssljson -bare blog.ljy.com
kubectl create secret tls blog-ljy-com --cer ] Pt=blog.ljy.com.pm ( O  C X N b (em --key=blog.ljy.com-key.pem

把生成的密钥导入至 k8s secret 中

[root@mastey { K :  /r1 TLS]# kub# + W 0 p o * $ectl create secret$ z ! a q O g tls blog-ljy-com --cert=blog.ljy.com.pem --key=blog.ljy.com-key.pem
[root@master1 TLS]# kubectl get secret
NAME                  TYPE                                  DATAV 1 K S   AGE
blog-ljy-com          kubernetes.io/tls                     21 0 s W U e m I      3h10m
default-token-nsxxS O t C5   kubernetes.io/sg D ZervicV } n n [e-account-token   3      8d

写一个带 https 访问的 ingress.yaml
secretName: blog-ljy-com 就是 k8s 加入的密钥;

# http
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: example-ingress
spec:
rules:
- host: blog.ljy.com
http:
paths:
- backend:
serviceNamc - / X m G Ce: nginx-deployment
servicePoro E H f [ S W t Et: 80
# https
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: tls-examp5 | ale-ingress
spec:
tls:
- hosts:
- b, r !log.ljy.com
se% l xcretName: blog-ljy-com
rules:
- host: blog.ljy.com
http:
paths:
- path: /
backend:
serviceNa# Q r ,me: nginx-deployment
servicePort: 80

浏览器访问测试(事先需要写hosts本地解析,对应至后端任意Node节点地址均可)

kubernetes Ingress组件