在K8S上部署EMQX企业版集群

EMQX 企业版技术支持发表于:2022年03月07日 15:07:54更新于:2022年04月27日 17:36:42

一、通过 Helm3 在 Kubernetes 上部署 EMQX 4.0 集群

Helm3 新特性

  • 移除了 Tiller

  • 不同的 namespace 可以使用相同的 Release Name

  • 简化模板对象 .Capabilities

  • 使用 JSONSchema 验证 charts 的 Values

  • 将 requirements.yaml合并到 Chart.yaml 中

  • helm install 时需要指定 Release Name,开启自动生成需要 --generate-name 参数

  • 支持 push 到远端 registry (如:harbor)

  • 移除 helm serve

  • 命令行变化(将原先的命令保留为别名 Aliases)

    • helm delete --> helm uninstall

    • helm inspect -> helm show

    • helm fetch -> helm pull

  • go 导入路径改变 k8s.io/helm --> helm.sh/helm

具体新特性可以参考 Helm 官方文档

Install Helm3

Helm3 提供了官方脚本简化了安装步骤, 可以执行 curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash 一键安装, 或者查看 Helm 官方文档 的详细说明

快速部署一个简单的 EMQX 集群

  • 添加 helm 仓库

    $ helm repo add emqx https://repos.emqx.io/charts$ helm repo update
  • 查询 EMQX

    helm search repo emqx
    NAME           CHART VERSION  APP VERSION DESCRIPTION
    emqx/emqx      v4.0.0         v4.0.0      A Helm chart for EMQX
    emqx/emqx-ee v4.0.0           v4.0.0      A Helm chart for EMQX
    emqx/kuiper    0.1.1          0.1.1       A lightweight IoT edge analytic software
  • 启动 EMQX 集群,设置 service.type=NodePort

$ helm install my-emqx emqx/emqx --set service.type=NodePort
  • 查看 EMQX 集群情况

    $ kubectl get pods
    NAME       READY  STATUS             RESTARTS  AGE
    my-emqx-0  1/1     Running   0          56s
    my-emqx-1  1/1     Running   0          40s
    my-emqx-2  1/1     Running   0          21s
    
    $ kubectl exec -it my-emqx-0 -- emqx_ctl cluster status
    Cluster status: #{running_nodes =>
                          ['my-emqx@my-emqx-0.my-emqx-headless.default.svc.cluster.local',                       
                          'my-emqx@my-emqx-1.my-emqx-headless.default.svc.cluster.local',                       
                          'my-emqx@my-emqx-2.my-emqx-headless.default.svc.cluster.local'],                  stopped_nodes => []}
  • 查看 EMQX service

    $ kubectl get svc
    NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                                      AGE
    my-emqx              NodePort       10.101.143.92    <none>        1883:32756/TCP,8883:31569/TCP,8081:30585/TCP,8083:31804/TCP,8084:30523/TCP,18083:31253/TCP   4m33s
    my-emqx-headless     ClusterIP      None             <none>        1883/TCP,8883/TCP,8081/TCP,8083/TCP,8084/TCP,18083/TCP                                       4m33s

可以看到 my-emqx 的 18083 端口对应的宿主机 IP 是 31539。(NodePort 在每次部署的时候都会变化,以实际部署时为准。)

  • 访问 Kubernetes 的任意一台节点 IP 的 31539 端口,输入默认用户名:admin,默认密码:public,登陆 EMQX dashboard。

  • 删除 EMQX 集群

    $ helm uninstall my-emqx
    release "my-emqx" uninstalled

部署一个持久化的 EMQX 集群

EMQX 通过 创建 PVC 资源挂载 /opt/emqx/data/mnesia 目录实现持久化 pods,在部署 EMQX 之前,用户需要部署 Haproxy 或 Nginx-PLUS 等负载均衡器,并自行在 Kubernetes 中创建 PVC 资源或是 Storage Classes 资源

  • 启动 EMQX 集群

    • 如果用户部署了 PVC 资源,那么设置 persistence.existingClaim=your_pv_name

      $ helm install my-emqx emqx/emqx --set persistence.enabled=true --set persistence.existingClaim=your_pv_name
    • 如果用户部署了 Storage Classes 资源,那么设置persistence.storageClass=your_storageClass_name

      $ helm install my-emqx emqx/emqx --set persistence.enabled=true --set persistence.storageClass=your_storageClass_name
  • 查看 EMQX 集群情况

    $ kubectl get pods
    NAME       READY  STATUS             RESTARTS  AGE
    my-emqx-0  1/1     Running   0          56
    smy-emqx-1  1/1     Running   0          40s
    my-emqx-2  1/1     Running   0          21s
    
    $ kubectl exec -it my-emqx-0 -- emqx_ctl cluster status
    Cluster status: #{running_nodes =>
                          ['my-emqx@my-emqx-0.my-emqx-headless.default.svc.cluster.local',                       
                          'my-emqx@my-emqx-1.my-emqx-headless.default.svc.cluster.local',                       
                          'my-emqx@my-emqx-2.my-emqx-headless.default.svc.cluster.local'],                  stopped_nodes => []}
  • 以 Storage Classes 为例,可以看到 PVC 资源已经成功的建立

    $ kubectl get pvc
    NAME                  STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    emqx-data-my-emqx-0   Bound     pvc-8094cd75-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            2m11s
    emqx-data-my-emqx-1   Bound     pvc-9325441d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            99s
    emqx-data-my-emqx-2   Bound     pvc-ad425e9d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            56s

    集群会将 EMQX 的 /opt/emqx/data/mnesia 目录挂载到 PVC 中,当 Pods 被重新调度之后,EMQX 会从 /opt/emqx/data/mnesia 目录中获取数据并恢复

  • 查看 EMQX 的 ClusterIP

    $ kubectl get svc
    NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                  AGE
    my-emqx              ClusterIP   10.100.205.13   <none>        1883/TCP,8883/TCP,8081/TCP,8083/TCP,8084/TCP,18083/TCP   26m
    my-emqx-headless     ClusterIP   None            <none>        1883/TCP,8883/TCP,8081/TCP,8083/TCP,8084/TCP,18083/TCP   26m

可以看到 my-emqx 的 ClusterIP 为 10.100.205.13 (ClusterIP 在每次部署的时候都会变化,以实际部署时为准。)

  • 将负载均衡监听的 URL 的 1883、8883、8081、8083、8084、18083 端口转发到 my-emqx 的 ClusterIP,如果有 TLS 连接的需要,推荐在负载均衡器终结 SSL 连接。客户端与负载均衡器之间 TLS 安全连接,LB 与 EMQX 之间普通 TCP 连接。

  • 访问 URL:18083,输入默认用户名:admin,默认密码:public,登陆 EMQX dashboard。

  • 使用 helm upgrade 命令可以轻松扩展 EMQX 集群,下面以增加 EMQX 节点为例展示 helm upgrade 命令

    # 将 EMQX 的节点数量变更为5个# 注意:EMQX 的节点数量建议为单数$ helm upgrade --set replicaCount=5 my-emqx emqx/emqx
    Release "my-emqx" has been upgraded. Happy Helming!
  $ kubectl get pods  NAME       READY  STATUS             RESTARTS  AGE
  my-emqx-0  1/1    Running            0         4m25s
  my-emqx-1  1/1    Running            0         4m14s
  my-emqx-2  1/1    Running            0         4m
  my-emqx-3  1/1    Running            0         31s
  my-emqx-4  1/1    Running            0         15s

  $ kubectl exec -it my-emqx-0 -- emqx_ctl cluster status  Cluster status: #{running_nodes =>
                        ['my-emqx@my-emqx-0.my-emqx-headless.default.svc.cluster.local',                         'my-emqx@my-emqx-1.my-emqx-headless.default.svc.cluster.local',                         'my-emqx@my-emqx-2.my-emqx-headless.default.svc.cluster.local',                         'my-emqx@my-emqx-3.my-emqx-headless.default.svc.cluster.local',                         'my-emqx@my-emqx-4.my-emqx-headless.default.svc.cluster.local'],
                    stopped_nodes => []}
  • 删除 EMQX 集群

    $ helm uninstall my-emqx
    release "my-emqx" uninstalled

注意:EMQX 集群删除掉之后 PVC 资源不会自动释放掉,以便恢复 EMQX,确认不需要恢复后需要手动删除 PVC 资源

  $ kubectl get pvc
  NAME                  STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
  emqx-data-my-emqx-0   Bound     pvc-8094cd75-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            84m
  emqx-data-my-emqx-1   Bound     pvc-9325441d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            84m
  emqx-data-my-emqx-2   Bound     pvc-ad425e9d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            83m
  emqx-data-my-emqx-3   Bound     pvc-b6c5a565-adbd-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            25m
  emqx-data-my-emqx-4   Bound     pvc-c626cafd-adbd-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            25m

  $ kubectl delete pvc emqx-data-my-emqx-0 emqx-data-my-emqx-1 emqx-data-my-emqx-2 emqx-data-my-emqx-3 emqx-data-my-emqx-4                    
  persistentvolumeclaim "emqx-data-my-emqx-0" deleted
  persistentvolumeclaim "emqx-data-my-emqx-1" deleted
  persistentvolumeclaim "emqx-data-my-emqx-2" deleted
  persistentvolumeclaim "emqx-data-my-emqx-3" deleted
  persistentvolumeclaim "emqx-data-my-emqx-4" deleted

部署 EMQX Edge 集群和 EMQX 企业版集群

EMQX Edge

部署 EMQX Edge 集群指定 image.repository=emqx/emqx-edge,其他设置与部署 EMQX 集群保持一致

$ helm install my-emqx-edge emqx/emqx --set image.repository=emqx/emqx$ kubectl get pods
NAME            READY   STATUS    RESTARTS   AGEmy-emqx-edge-0  1/1     Running   0          35smy-emqx-edge-1  1/1     Running   0          23smy-emqx-edge-2  1/1     Running   0          9s

EMQX EE

部署 EMQX 企业版集群首先需要前往 www.emqx.com 申请并下载 License 文件,并将 License 文件创建为 Secret 资源

$ kubectl create secret generic your-license-secret-name --from-file=/path/to/emqx.lic

然后在部署时指定 repo 为 emqx/emqx-eeemqxLicneseSecretName=your-license-secret-name, 其他设置与部署 EMQX 集群保持一致

$ helm install my-emqx-ee emqx/emqx-ee emqxLicneseSecretName=your-license-secret-name

EMQX Helm Chart 配置项

参数描述Default Value
replicaCountEMQX 节点数量,建议保持奇数个节点,不然脑裂后无法自动恢复3
image.repositoryEMQX 镜像名称emqx/emqx
image.pullPolicy获取镜像的策略IfNotPresent
persistence.enabled是否启用 PVCfalse
persistence.storageClassStorage class 名称nil
persistence.existingClaimPV 名称""
persistence.accessModePVC 访问模式ReadWriteOnce
persistence.sizePVC 容量20Mi
resourcesCPU/ 内存资源{}
nodeSelectorpod 分配的节点标签{}
tolerations
[]
affinity
{}
service.typeEmqx cluster service typeClusterIP
emqxConfigEMQX 配置项,详情查看文档{}
emqxLicneseSecretNameEMQX 企业版需要手动将 License 文件创建为 Secret 资源 (仅在 emqx/emqx-e 有效)""

当需要设置复杂参数的时候,可以使用 Yaml 文件来记录参数

$ helm install my-emqx emqx/emqx -f values.yaml

你可以从 Github 获取默认的 values.yaml

地址:https://github.com/emqx/emqx-rel/tree/master/deploy/charts/emqx

二、通过原生yaml方式部署EMQX集群

首先,可以从方法一GitHub地址去获取默认的yaml文件,再通过修改想设置的参数去实现emqx集群部署。例如:

1.编写emqx.yaml

---
# Source: emqx-ee/templates/rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: emqx-ee
  name: emqx-ee
---
# Source: emqx-ee/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: emqx-ee-env
  namespace: emqx-ee
  labels:
    app.kubernetes.io/name: emqx-ee
    
data:
    EMQX_CLUSTER__K8S__ADDRESS_TYPE: "hostname"
    EMQX_CLUSTER__K8S__APISERVER: "https://kubernetes.default.svc:443"
    EMQX_CLUSTER__K8S__SUFFIX: "svc.cluster.local"
    EMQX_LOG__TO: "both"
    
---# Source: emqx-ee/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: emqx-ee-acl
  namespace: emqx-ee
  labels:
    app.kubernetes.io/name: emqx-ee
    
data:
  "acl.conf": >   
      {allow, {user, "dashboard"}, subscribe, ["$SYS/#"]}. 
      {allow, {ipaddr, "127.0.0.1"}, pubsub, ["$SYS/#", "#"]}. 
      {deny, all, subscribe, ["$SYS/#", {eq, "#"}]}. 
      
---
# Source: emqx-ee/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: emqx-ee-loaded-plugins
  namespace: emqx-ee
  labels:
    app.kubernetes.io/name: emqx-ee
  
data:
  "loaded_plugins": |   
    {emqx_management, true}. {emqx_dashboard, true}. {emqx_rule_engine, true}. {emqx_modules, true}.
---
# Source: emqx-ee/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: emqx-ee-loaded-modules
  namespace: emqx-ee
  labels:
    app.kubernetes.io/name: emqx-ee
  
data:
  "loaded_modules": |     
      [ { "name": "internal_acl", "enable": true, "configs": {"acl_rule_file": "etc/acl.conf"} }, { "name": "presence", "enable": true, "configs": {"qos": 0} }, { "name": "recon", "enable": true, "configs": {} }, { "name": "retainer", "enable": true, "configs": { "expiry_interval": 0, "max_payload_size": "1MB", "max_retained_messages": 0, "storage_type": "ram" } } ]
---
# Source: emqx-ee/templates/rbac.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: emqx-ee
  name: emqx-ee
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - get
  - watch
  - list
---
# Source: emqx-ee/templates/rbac.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: emqx-ee
  name: emqx-ee
subjects:
  - kind: ServiceAccount
    name: emqx-ee
    namespace: emqx-ee
roleRef:
  kind: Role
  name: emqx-ee
  apiGroup: rbac.authorization.k8s.io
---
# Source: emqx-ee/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: emqx-ee
  namespace: emqx-ee
  labels:
    app.kubernetes.io/name: emqx-ee
    
spec:
  type: NodePort
  ports:
  - name: mqtt
    port: 1883
    protocol: TCP
    targetPort: mqtt
    nodePort: null
  - name: mqttssl
    port: 8883
    protocol: TCP
    targetPort: mqttssl
    nodePort: null
  - name: mgmt
    port: 8081
    protocol: TCP
    targetPort: mgmt
    nodePort: null
  - name: ws
    port: 8083
    protocol: TCP
    targetPort: ws
    nodePort: null
  - name: wss
    port: 8084
    protocol: TCP
    targetPort: wss
    nodePort: null
  - name: dashboard
    port: 18083
    protocol: TCP
    targetPort: dashboard
    nodePort: null
  selector:
    app.kubernetes.io/name: emqx-ee
---
# Source: emqx-ee/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: emqx-ee-headless
  namespace: emqx-ee
  labels:
    app.kubernetes.io/name: emqx-ee
    
spec:
  type: ClusterIP
  publishNotReadyAddresses: true
  sessionAffinity: None
  clusterIP: None
  ports:
  - name: mqtt
    port: 1883
    protocol: TCP
    targetPort: mqtt
  - name: mqttssl
    port: 8883
    protocol: TCP
    targetPort: mqttssl
  - name: mgmt
    port: 8081
    protocol: TCP
    targetPort: mgmt
  - name: ws
    port: 8083
    protocol: TCP
    targetPort: ws
  - name: wss
    port: 8084
    protocol: TCP
    targetPort: wss
  - name: dashboard
    port: 18083
    protocol: TCP
    targetPort: dashboard
  - name: ekka
    port: 4370
    protocol: TCP
    targetPort: ekka
  selector:
    app.kubernetes.io/name: emqx-ee
---
# Source: emqx-ee/templates/StatefulSet.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: emqx-ee
  namespace: emqx-ee
  labels:
    app.kubernetes.io/name: emqx-ee
    
spec:
  serviceName: emqx-ee-headless
  podManagementPolicy: Parallel
  
  updateStrategy:
    type: RollingUpdate
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: emqx-ee
      
  template:
    metadata:
      labels:
        app: emqx-ee
        version: 4.3.5
        app.kubernetes.io/name: emqx-ee
        
    spec:
      volumes:
      - name: emqx-loaded-plugins
        configMap:
          name: emqx-ee-loaded-plugins
          items:
          - key: loaded_plugins
            path: loaded_plugins
      - name: emqx-loaded-modules
        configMap:
          name: emqx-ee-loaded-modules
          items:
          - key: loaded_modules
          path: loaded_modules
      - name: emqx-acl
        configMap:
          name: emqx-ee-acl
          items:
          - key: acl.conf
            path: acl.conf
      - name: emqx-license
        secret:
          secretName: emqx-license
      - name: host-time
        hostPath:
          path: /etc/localtime
      serviceAccountName:  emqx-ee
      securityContext:
        fsGroup: 1000
        fsGroupChangePolicy: Always
        runAsUser: 1000
        supplementalGroups:
        - 1000
      containers:
        - name: emqx
          image: "emqx/emqx-ee:4.3.5"
          imagePullPolicy: IfNotPresent
          securityContext:
            runAsNonRoot: true
            runAsUser: 1000
          ports:
          - name: mqtt
            containerPort: 1883
          - name: mqttssl
            containerPort: 8883
          - name: mgmt
            containerPort: 8081
          - name: ws
            containerPort: 8083
          - name: wss
            containerPort: 8084
          - name: dashboard
            containerPort: 18083
          - name: ekka
            containerPort: 4370
          envFrom:
            - configMapRef:
                name: emqx-ee-env
          env:
          - name: EMQX_NAME
            value: emqx
          - name: EMQX_CLUSTER__K8S__APP_NAME
            value: emqx
          - name: EMQX_CLUSTER__DISCOVERY
            value: k8s
          - name: EMQX_CLUSTER__K8S__SERVICE_NAME
            value: emqx-ee-headless
          - name: EMQX_CLUSTER__K8S__NAMESPACE
            value: emqx-ee
          resources:
            limits:
              cpu: 500m
              memory: 512Mi
            requests:
              cpu: 500m
              memory: 512Mi
            volumeMounts:
            - name: emqx-acl
              mountPath: "/opt/emqx/etc/acl.conf"
              subPath: "acl.conf"
            - name: emqx-loaded-plugins
              mountPath: "/opt/emqx/data/loaded_plugins"
              subPath: "loaded_plugins"
            - name: emqx-loaded-modules
              mountPath: "/opt/emqx/data/loaded_modules"
              subPath: "loaded_modules"
            - name: emqx-license
              mountPath: "/opt/emqx/etc/emqx.lic"
              subPath: "emqx.lic"
              readOnly: true
            - name: host-time
              mountPath: /etc/localtime
            readinessProbe:
              httpGet:
                path: /status
                port: 8081
              initialDelaySeconds: 5
              periodSeconds: 5

2.创建namespace

kubectl create ns emqx-ee

3.创建secret(事先需要准备好相应的license文件)

kubectl create secret generic emqx-license --from-file=./emqx.lic -n emqx-ee

4.通过应用emqx.yaml文件创建emqx集群

kubectl apply -f emqx.yaml -n emqx-ee

5.在创建完成后,查看pod状态以及service

[root@chq ~]# kubectl get po -n emqx-ee
NAME        READY   STATUS    RESTARTS   AGE
emqx-ee-0   1/1     Running   0          22s
emqx-ee-1   1/1     Running   0          22s
emqx-ee-2   1/1     Running   0          22s
[root@chq ~]# kubectl get svc -n emqx-ee
NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                                                                      AGE
emqx-ee            NodePort    10.233.50.73   <none>        1883:30674/TCP,8883:30855/TCP,8081:30793/TCP,8083:31601/TCP,8084:31692/TCP,18083:32238/TCP   29s
emqx-ee-headless   ClusterIP   None           <none>        1883/TCP,8883/TCP,8081/TCP,8083/TCP,8084/TCP,18083/TCP,4370/TCP                              29s


    您需要登录后才可以回复