kubernetes学习四-kubernetes配置NFS持久化存储

发表于 | 更新于: | 分类于 | 次阅读
字数统计: 3.1k | 阅读时长 ≈ 15

1 环境

k8s集群基于kubeadm工具部署，过程参考《kubernetes学习二-kubeadm安装kubernetes集群-单master》

kubernetes版本：1.23.3

主机 IP 角色
node1 192.168.1.201 master
node2 192.168.1.202 node
node3 192.168.1.203 nfs server

2 介绍

image-20220608171554052

image-20220608171226158

PV说明：

  • 由集群管理员创建和管理
  • 设置底层存储，比如使用NFS
  • 设置存储容量大小
  • 设置访问模式（Access mode）
  • 指定Reclaim Policy
  • PV不属于任何命名空间，是节点层面的资源
  • PV的状态：Available -可用，未绑定PVC；Bound - 已绑定PVC；Terminating -删除中；

PVC说明：

  • 由开发团队创建和管理
  • 不需要知道底层存储的实现细节
  • 指定所需的最低存储容量
  • 指定访问模式
  • 可选指定PV名称volumeName,不指定时自动由kuberntes自动匹配（根据存储容量和访问模式）
  • PVC和PV的关系是一对一，只能将一个PVC绑定到一个PV上
  • PVC属于某个命名空间
  • PVC状态：Pending - 待绑定；Bound- 已绑定PVC；Terminatinig-删除中

3 准备NFS服务器

1
2
3
4
5
6
7
8
9
10
11
12
13
yum -y install nfs-utils rpcbind
systemctl  start rpcbind.service 
systemctl  start nfs

#配置, * 代表所有IP
mkdir /mnt/nfs -p
chown nfsnobody.nfsnobody /data/nfs

cat>>/etc/exports<<EOF
/mnt/nfs *(rw,sync,no_root_squash,no_all_squash)
EOF
# 加载配置
exportfs  -arv

所有kubernetes集群节点安装nfs-utils

1
yum install -y  nfs-utils

kubernetes集群节点验证nfs服务器挂载点

1
2
3
[root@node1 nfs]# showmount -e 192.168.1.203
Export list for 192.168.1.203:
/mnt/nfs *

4 NFS静态供给（Static Provision）

我们提前创建PV，然后通过PVC申请PV并在pod中使用，这种方式叫做静态供给（Static Provision）。

4.1 首先创建PV（PersistentVolume）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-nfs-pv1               
spec:
  capacity:
    storage: 5Mi                 
  accessModes:  
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle        
  storageClassName: nfs                         
  mountOptions:
    - hard
    - nfsvers=4.1
  nfs:
    path: /opt/nfs/pv1                            # 指定PV在NFS服务器上的对应目录。
    server: 192.168.0.75                          # NFS服务器的地址
  1. name: 定义PV的名称。
  2. capacity: 指定存储的容量
  3. accessModes: 指定访问模式，支持三种模式。ReadWriteOnce标识PV能以read-write模式mount到单个节点，ReadOnlyMany表示PV能以read-only模式mount到多个节点，ReadWriteMany表示PV能以read-write模式mount到多个节点。
  4. persistentVolumeReclaimPolicy: 指定回收策略。Retain表示需要管理员手工回收即手动删除文件；Recycle表示清除PV中的数据即删除PVC时直接删除PV中的数据；Delete表示删除Storage Provider上的对应存储资源，例如 Aws、 EBS、 GCE PD、Azure Disk、OpenStack Cinder Volume等
  5. storageClassName: 指定PV的class为nfs，相当于设置PV的分类。PVC可以指定class申请相应class的PV。
  6. nfs: 指定PV在NFS服务器上的路径
1
2
3
4
5
6
7
# 创建PV
[root@k8s-master nfs_]# kuberctl apply -f pv.yaml
# 查看PV
[root@k8s-master nfs_]# kubectl get pv
NAME         CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
my-nfs-pv1   5Mi        RWO            Recycle          Available           nfs                     92m
# STATUS 为Available,表示就绪，可以被PVC申请。

4.2 创建PVC（PersistentVolumeClaim）

PVC创建就很简单，只需要指定PV的容量、访问模式、class即可。

1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc1-pv1                    # 指定pvc名称
  namespace: geocms                 # 指定PVC属于哪个命名空间
spec:
  resources:                     
    requests:
      storage: 5Mi                  # 指定PV最低存储容量
  accessModes:
    - ReadWriteOnce                 # 指定访问模式
  storageClassName: nfs             # 指定匹配PV的class
1
2
3
4
5
6
7
8
9
10
11
12
13
# 创建PVC
[root@k8s-master nfs_]# kubectl apply -f nfs_pvc1.yaml 
persistentvolumeclaim/pvc1-pv1 created
# 查看PVC
# STATUS 为Bound 表示申请成功
[root@k8s-master nfs_]# kubectl get pvc -n geocms 
NAME       STATUS   VOLUME       CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc1-pv1   Bound    my-nfs-pv1   5Mi        RWO            nfs            7s
# 查看PV
# STATUS为Bound，CLATMIM表示被geocms命名空间的PVC(pvc1-pv1)绑定成功。 
[root@k8s-master nfs_]# kubectl get pv
NAME         CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM             STORAGECLASS   REASON   AGE
my-nfs-pv1   5Mi        RWO            Recycle          Bound    geocms/pvc1-pv1   nfs                     106m

4.3 POD挂载

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: geocms                      # 要和PVC在同一个命名空间
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.20.1
        volumeMounts:
        - name: nginx-html
          mountPath: /usr/share/nginx/html
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 80
      volumes:
      - name: nginx-html
        persistentVolumeClaim:             # 指定存储类型
          claimName: pvc1-pv1              # 指定PVC的名称
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: geocms
spec:
  type: NodePort
  selector:
    app: nginx
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30001
1
2
3
4
# 创建Deployment
kubectl apply -f nginx.yaml
# 在/opt/nfs/pv1下创建index.html文件
# 访问nginx，显示成功表示挂载成功。

2 NFS动态供给（Static Provision）

参考：https://www.cnblogs.com/zhangb8042/p/14252294.html

动态供给，即如果没有满足PVC条件的PV，会动态创建PV。相比静态供给，动态供给由明显的优势：不需要提前创建按PV,减少了管理员的工作量，效率高。

动态供给是通过StorageClass实现的，StorageClass定义了如何创建PV，下面给出NFS实现动态供给的例子。

github地址: https://github.com/kubernetes-retired/external-storage

2.1 下载yaml配置

https://github.com/kubernetes-retired/external-storage/tree/master/nfs

1
2
3
wget https://raw.githubusercontent.com/kubernetes-retired/external-storage/master/nfs-client/deploy/rbac.yaml
wget https://raw.githubusercontent.com/kubernetes-retired/external-storage/master/nfs-client/deploy/class.yaml
wget https://raw.githubusercontent.com/kubernetes-retired/external-storage/master/nfs-client/deploy/deployment.yaml

2.2 创建StorageClass

1
kubectl  apply -f class.yaml
1
2
3
4
5
6
7
8
9
10
# class.yaml
# 动态供给是通过StorageClass实现的，StorageClass定义如何创建PV。
---------------------------------------------------------------
apiVersion: storage.k8s.io/v1
kind: StorageClass              
metadata:
  name: managed-nfs-storage     
provisioner: fuseim.pri/ifs     # 可自定义名称，但必须和deployment配置的变量 “PROVISIONER_NAME” 匹配
parameters:
  archiveOnDelete: "false"

2.3 创建 ClusterRole

创建ClusterRole、ClusterRoleBinding,Role和RoleBinding（如果您在集群上使用 RBAC 授权，这是必需的，这是较新 kubernetes 版本的默认设置）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
# rbac.yaml
----------------------------------------------------------------
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: default
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

2.4 创建Deployment

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# deployment.yaml 
------------------------------------------------------------------
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: default
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME    
              value: fuseim.pri/ifs           # PROVISIONER_NAME变量的值必须好class中的 “provisioner” 匹配
            - name: NFS_SERVER
              value: 192.168.1.203            # nfs服务器ip
            - name: NFS_PATH
              value: /mnt/nfs                 # nfs服务器的挂载目录
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.1.2036            # nfs服务器ip
            path: /mnt/nfs                    # nfs服务器的挂载目录
1
2
3
4
5
6
7
8
9
10
11
12
# 创建Deployment
[root@master ~]# kubectl apply -f deployment.yaml

# 查看pod
[root@master ~]# kubectl get pod 
NAME                                     READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-dfb75c8bb-42p8l   1/1     Running   0          80s

# 查看存储类 kubectl get storageclasses.storage.k8s.io
[root@master ~]# kubectl get sc        
NAME                  PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
managed-nfs-storage   fuseim.pri/ifs   Delete          Immediate           false                  5m56s

2.5 设置默认存储

设置managed-nfs-storage为kubernetes的默认存储后端

  • 我们可以用kubectl patch命令来更新
1
2
3
4
# 设置managed-nfs-storage为默认存储
kubectl patch storageclass managed-nfs-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
# 取消设置managed-nfs-storage为默认存储
kubectl patch storageclass managed-nfs-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
  • 也可以修改class.yaml文件

增加如下2行，设置为默认存储类

1
2
3
4
5
6
7
8
9
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: managed-nfs-storage
  annotations:                                            # 配置默认存储类
    storageclass.kubernetes.io/is-default-class: "true"   # 配置默认存储类
provisioner: nfsserver-test # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "false"

验证

1
2
3
[root@node1 nfs]# kubectl get sc
NAME                            PROVISIONER      RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
managed-nfs-storage (default)   nfsserver-test   Delete          Immediate           false                  160m

2.6 修改api-server的配置

修改/etc/kubernetes/manifests/kube-apiserver.yaml 文件

添加添加- –feature-gates=RemoveSelfLink=false

1
2
3
4
5
6
7
[root@master ~]# grep -B 5 'feature-gates' /etc/kubernetes/manifests/kube-apiserver.yaml
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --feature-gates=RemoveSelfLink=false #添加内容

修改后 apiserver会自动重启。

2.7 测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# test-pvc.yaml
---------------------------------------------------------------
[root@master ~]# cat  test-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: geocms-html-claim
  annotations:
    volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 创建PVC
[root@master ~]# kubectl  apply -f test-pvc.yaml
persistentvolumeclaim/test-claim created


# 查看PVC。STATUS 为Bound表示绑定PV成功，VOLUME为挂载的PV名称
[root@master ~]# kubectl get pvc
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
geocms-html-claim   Bound    pvc-93404728-f6f4-4ba2-8146-1553be9bedad   1Gi        RWX            managed-nfs-storage   12s

# 查看PV。发现自动创建了PV，STATUS为Bound，表示绑定成功，被default命名空间的geocms-html-claim申请成功。
[root@k8s-master geocms]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                       STORAGECLASS          REASON   AGE
pvc-93404728-f6f4-4ba2-8146-1553be9bedad   1Gi        RWX            Delete           Bound    default/geocms-html-claim   managed-nfs-storage            84s

3 问题汇总

3.1 pvc一直Pengding

现象：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# pvc状态
[root@node1 nfs]# kubectl get pvc
NAME         STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS          AGE
test-claim   Pending                                      managed-nfs-storage   3s

# pvc信息
[root@node1 nfs]# kubectl describe pvc test-claim 
Name:          test-claim
Namespace:     default
StorageClass:  managed-nfs-storage
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-class: managed-nfs-storage
               volume.beta.kubernetes.io/storage-provisioner: nfsserver-test
               volume.kubernetes.io/storage-provisioner: nfsserver-test
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type    Reason                Age               From                         Message
  ----    ------                ----              ----                         -------
  Normal  ExternalProvisioning  7s (x3 over 26s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "nfsserver-test" or manually created by system administrator
[root@node1 nfs]#

原因：

  1. 需修改api-server.yaml配置，添加- –feature-gates=RemoveSelfLink=false

  2. nfs挂载目录需要有权限

  3. nfs存储类没有配置正确。（我遇到的问题是nfs的depoyment无法正常连接api-server，原因是集群初始化时候flannel的网络配置和默认网络不一致。）

解决办法：

参考 3.2 nfs的deployment异常

3.2 nfs的deployment异常

参考： Kubeadm 部署 使用flannel无法连接service/kubernetes

现象：deployment创建的pod总是异常挂掉或无法正常启动。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# nfs deploy状态
[root@node1 nfs]# kubectl get pod
NAME                                     READY   STATUS              RESTARTS   AGE
nfs-client-provisioner-b8b468858-m9c79   0/1     ContainerCreating   0          4m24s

# 查看pod状态
[root@node1 nfs]# kubectl describe pod nfs-client-provisioner-b8b468858-m9c79 
......
  Normal   SandboxChanged          20s (x12 over 31s)   kubelet            Pod sandbox changed, it will be killed and re-created.
  Warning  FailedCreatePodSandBox  19s (x4 over 22s)    kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "01a249d53627976e4d872e44004b436600bef233a5dee8814760e9e0cdabc015" network for pod "nfs-client-provisioner-b8b468858-m9c79": networkPlugin cni failed to set up pod "nfs-client-provisioner-b8b468858-m9c79_default" network: failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 101.100.1.1/24

# 查看pod日志
[root@node1 nfs]# kubectl get pod
NAME                                     READY   STATUS              RESTARTS   AGE
nfs-client-provisioner-b8b468858-m9c79   0/1     ContainerCreating   0          4m24s
1
2
3
4
5
6
7
# 查看各节点flannel网络配置
[root@node1 nfs]# cat /run/flannel/subnet.env 
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=101.100.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
[root@node1 nfs]#

原因：flanenl网络地址端配置不正确，FLANNEL_NETWORK和FLANNEL_SUBNET需要在一个网段。

解决办法：

flannel网络的FLANNEL_NETWORK和FLANNEL_SUBNET需要在一个网段，使flannel的默认网段和集群初始化的cni网络在一个网段。重新安装kubernetes 使用默认–pod-network-cidr=10.244.0.0/16 （下面示例均使用101.100.0.0/16）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# 查看flannel.yaml,发现flannel默认网络是10.244.0.0/16
[root@node1 home]# cat flannel.yml
.....
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
.....

# 配置集群初始化yaml文件，设置podSubnet:10.244.0.0/16(也可以配置为其他段，只需要保证和flannel.yaml中的网段即`FLANNEL_NETWORK`和`FLANNEL_SUBNET`需要在一个网段。)
# 这里我直接修改flannel.yaml和初始化yaml文件内的网段均为101.100.0.0/16
# 重新初始化安装kubernentes
[root@node1 nfs]# cat /run/flannel/subnet.env 
FLANNEL_NETWORK=101.100.0.0/16
FLANNEL_SUBNET=101.100.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
[root@node1 nfs]# 

# 查看nfs pod和pvc均正常了
[root@node1 nfs]# kubectl describe pod nfs-client-provisioner-b8b468858-gn2b6 
......
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  15s   default-scheduler  Successfully assigned default/nfs-client-provisioner-b8b468858-gn2b6 to node2
  Normal  Pulling    14s   kubelet            Pulling image "quay.io/external_storage/nfs-client-provisioner:latest"
  Normal  Pulled     10s   kubelet            Successfully pulled image "quay.io/external_storage/nfs-client-provisioner:latest" in 4.239848452s
  Normal  Created    10s   kubelet            Created container nfs-client-provisioner
  Normal  Started    10s   kubelet            Started container nfs-client-provisioner

[root@node1 nfs]# kubectl get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
test-claim   Bound    pvc-811d031b-e842-4103-bdaa-cc89f51e1fc8   1Mi        RWX            managed-nfs-storage   2m18s

# pvc状态
[root@node1 nfs]# kubectl describe pvc test-claim 
......
Events:
  Type    Reason                 Age                  From                                                                                        Message
  ----    ------                 ----                 ----                                                                                        -------
  Normal  ExternalProvisioning   87s (x6 over 2m13s)  persistentvolume-controller                                                                 waiting for a volume to be created, either by external provisioner "nfsserver-test" or manually created by system administrator
  Normal  ExternalProvisioning   7s                   persistentvolume-controller                                                                 waiting for a volume to be created, either by external provisioner "nfsserver-test" or manually created by system administrator
  Normal  Provisioning           5s                   nfsserver-test_nfs-client-provisioner-b8b468858-gn2b6_bb6b9b33-8e34-11ec-ab4a-ca0aed401744  External provisioner is provisioning volume for claim "default/test-claim"
  Normal  ProvisioningSucceeded  5s                   nfsserver-test_nfs-client-provisioner-b8b468858-gn2b6_bb6b9b33-8e34-11ec-ab4a-ca0aed401744  Successfully provisioned volume pvc-811d031b-e842-4103-bdaa-cc89f51e1fc8
-------------本文结束感谢您的阅读-------------
坚持原创技术分享，您的支持将鼓励我继续创作！