一、通过 Helm3 在 Kubernetes 上部署 EMQX 4.0 集群
Helm3 新特性
移除了 Tiller
不同的 namespace 可以使用相同的 Release Name
简化模板对象
.Capabilities使用
JSONSchema验证 charts 的 Values将
requirements.yaml合并到Chart.yaml中helm install 时需要指定 Release Name，开启自动生成需要
--generate-name参数支持 push 到远端 registry （如：harbor）
移除 helm serve
命令行变化（将原先的命令保留为别名 Aliases）
helm delete-->helm uninstallhelm inspect->helm showhelm fetch->helm pull
go 导入路径改变
k8s.io/helm-->helm.sh/helm
具体新特性可以参考 Helm 官方文档
Install Helm3
Helm3 提供了官方脚本简化了安装步骤, 可以执行 curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash 一键安装, 或者查看 Helm 官方文档 的详细说明
快速部署一个简单的 EMQX 集群
添加 helm 仓库
$ helm repo add emqx https://repos.emqx.io/charts$ helm repo update
查询 EMQX
helm search repo emqx NAME CHART VERSION APP VERSION DESCRIPTION emqx/emqx v4.0.0 v4.0.0 A Helm chart for EMQX emqx/emqx-ee v4.0.0 v4.0.0 A Helm chart for EMQX emqx/kuiper 0.1.1 0.1.1 A lightweight IoT edge analytic software
启动 EMQX 集群，设置
service.type=NodePort
$ helm install my-emqx emqx/emqx --set service.type=NodePort
查看 EMQX 集群情况
$ kubectl get pods NAME READY STATUS RESTARTS AGE my-emqx-0 1/1 Running 0 56s my-emqx-1 1/1 Running 0 40s my-emqx-2 1/1 Running 0 21s $ kubectl exec -it my-emqx-0 -- emqx_ctl cluster status Cluster status: #{running_nodes => ['my-emqx@my-emqx-0.my-emqx-headless.default.svc.cluster.local', 'my-emqx@my-emqx-1.my-emqx-headless.default.svc.cluster.local', 'my-emqx@my-emqx-2.my-emqx-headless.default.svc.cluster.local'], stopped_nodes => []}查看 EMQX service
$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE my-emqx NodePort 10.101.143.92 <none> 1883:32756/TCP,8883:31569/TCP,8081:30585/TCP,8083:31804/TCP,8084:30523/TCP,18083:31253/TCP 4m33s my-emqx-headless ClusterIP None <none> 1883/TCP,8883/TCP,8081/TCP,8083/TCP,8084/TCP,18083/TCP 4m33s
可以看到 my-emqx 的 18083 端口对应的宿主机 IP 是 31539。（NodePort 在每次部署的时候都会变化，以实际部署时为准。）
访问 Kubernetes 的任意一台节点 IP 的 31539 端口，输入默认用户名：admin，默认密码：public，登陆 EMQX dashboard。
删除 EMQX 集群
$ helm uninstall my-emqx release "my-emqx" uninstalled
部署一个持久化的 EMQX 集群
EMQX 通过 创建 PVC 资源挂载 /opt/emqx/data/mnesia 目录实现持久化 pods，在部署 EMQX 之前，用户需要部署 Haproxy 或 Nginx-PLUS 等负载均衡器，并自行在 Kubernetes 中创建 PVC 资源或是 Storage Classes 资源
启动 EMQX 集群
如果用户部署了 PVC 资源，那么设置
persistence.existingClaim=your_pv_name$ helm install my-emqx emqx/emqx --set persistence.enabled=true --set persistence.existingClaim=your_pv_name
如果用户部署了 Storage Classes 资源，那么设置
persistence.storageClass=your_storageClass_name$ helm install my-emqx emqx/emqx --set persistence.enabled=true --set persistence.storageClass=your_storageClass_name
查看 EMQX 集群情况
$ kubectl get pods NAME READY STATUS RESTARTS AGE my-emqx-0 1/1 Running 0 56 smy-emqx-1 1/1 Running 0 40s my-emqx-2 1/1 Running 0 21s $ kubectl exec -it my-emqx-0 -- emqx_ctl cluster status Cluster status: #{running_nodes => ['my-emqx@my-emqx-0.my-emqx-headless.default.svc.cluster.local', 'my-emqx@my-emqx-1.my-emqx-headless.default.svc.cluster.local', 'my-emqx@my-emqx-2.my-emqx-headless.default.svc.cluster.local'], stopped_nodes => []}以 Storage Classes 为例，可以看到 PVC 资源已经成功的建立
$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE emqx-data-my-emqx-0 Bound pvc-8094cd75-adb5-11e9-80cc-0697b59e8064 1Gi RWO gp2 2m11s emqx-data-my-emqx-1 Bound pvc-9325441d-adb5-11e9-80cc-0697b59e8064 1Gi RWO gp2 99s emqx-data-my-emqx-2 Bound pvc-ad425e9d-adb5-11e9-80cc-0697b59e8064 1Gi RWO gp2 56s
集群会将 EMQX 的
/opt/emqx/data/mnesia目录挂载到 PVC 中，当 Pods 被重新调度之后，EMQX 会从/opt/emqx/data/mnesia目录中获取数据并恢复查看 EMQX 的 ClusterIP
$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE my-emqx ClusterIP 10.100.205.13 <none> 1883/TCP,8883/TCP,8081/TCP,8083/TCP,8084/TCP,18083/TCP 26m my-emqx-headless ClusterIP None <none> 1883/TCP,8883/TCP,8081/TCP,8083/TCP,8084/TCP,18083/TCP 26m
可以看到 my-emqx 的 ClusterIP 为 10.100.205.13 （ClusterIP 在每次部署的时候都会变化，以实际部署时为准。）
将负载均衡监听的 URL 的 1883、8883、8081、8083、8084、18083 端口转发到
my-emqx的 ClusterIP，如果有 TLS 连接的需要，推荐在负载均衡器终结 SSL 连接。客户端与负载均衡器之间 TLS 安全连接，LB 与 EMQX 之间普通 TCP 连接。访问
URL:18083，输入默认用户名：admin，默认密码：public，登陆 EMQX dashboard。使用
helm upgrade命令可以轻松扩展 EMQX 集群，下面以增加 EMQX 节点为例展示helm upgrade命令# 将 EMQX 的节点数量变更为5个# 注意：EMQX 的节点数量建议为单数$ helm upgrade --set replicaCount=5 my-emqx emqx/emqx Release "my-emqx" has been upgraded. Happy Helming!
$ kubectl get pods NAME READY STATUS RESTARTS AGE
my-emqx-0 1/1 Running 0 4m25s
my-emqx-1 1/1 Running 0 4m14s
my-emqx-2 1/1 Running 0 4m
my-emqx-3 1/1 Running 0 31s
my-emqx-4 1/1 Running 0 15s
$ kubectl exec -it my-emqx-0 -- emqx_ctl cluster status Cluster status: #{running_nodes =>
['my-emqx@my-emqx-0.my-emqx-headless.default.svc.cluster.local', 'my-emqx@my-emqx-1.my-emqx-headless.default.svc.cluster.local', 'my-emqx@my-emqx-2.my-emqx-headless.default.svc.cluster.local', 'my-emqx@my-emqx-3.my-emqx-headless.default.svc.cluster.local', 'my-emqx@my-emqx-4.my-emqx-headless.default.svc.cluster.local'],
stopped_nodes => []}删除 EMQX 集群
$ helm uninstall my-emqx release "my-emqx" uninstalled
注意：EMQX 集群删除掉之后 PVC 资源不会自动释放掉，以便恢复 EMQX，确认不需要恢复后需要手动删除 PVC 资源
$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE emqx-data-my-emqx-0 Bound pvc-8094cd75-adb5-11e9-80cc-0697b59e8064 1Gi RWO gp2 84m emqx-data-my-emqx-1 Bound pvc-9325441d-adb5-11e9-80cc-0697b59e8064 1Gi RWO gp2 84m emqx-data-my-emqx-2 Bound pvc-ad425e9d-adb5-11e9-80cc-0697b59e8064 1Gi RWO gp2 83m emqx-data-my-emqx-3 Bound pvc-b6c5a565-adbd-11e9-80cc-0697b59e8064 1Gi RWO gp2 25m emqx-data-my-emqx-4 Bound pvc-c626cafd-adbd-11e9-80cc-0697b59e8064 1Gi RWO gp2 25m $ kubectl delete pvc emqx-data-my-emqx-0 emqx-data-my-emqx-1 emqx-data-my-emqx-2 emqx-data-my-emqx-3 emqx-data-my-emqx-4 persistentvolumeclaim "emqx-data-my-emqx-0" deleted persistentvolumeclaim "emqx-data-my-emqx-1" deleted persistentvolumeclaim "emqx-data-my-emqx-2" deleted persistentvolumeclaim "emqx-data-my-emqx-3" deleted persistentvolumeclaim "emqx-data-my-emqx-4" deleted
部署 EMQX Edge 集群和 EMQX 企业版集群
EMQX Edge
部署 EMQX Edge 集群指定 image.repository=emqx/emqx-edge，其他设置与部署 EMQX 集群保持一致
$ helm install my-emqx-edge emqx/emqx --set image.repository=emqx/emqx$ kubectl get pods NAME READY STATUS RESTARTS AGEmy-emqx-edge-0 1/1 Running 0 35smy-emqx-edge-1 1/1 Running 0 23smy-emqx-edge-2 1/1 Running 0 9s
EMQX EE
部署 EMQX 企业版集群首先需要前往 www.emqx.com 申请并下载 License 文件，并将 License 文件创建为 Secret 资源
$ kubectl create secret generic your-license-secret-name --from-file=/path/to/emqx.lic
然后在部署时指定 repo 为 emqx/emqx-ee, emqxLicneseSecretName=your-license-secret-name, 其他设置与部署 EMQX 集群保持一致
$ helm install my-emqx-ee emqx/emqx-ee emqxLicneseSecretName=your-license-secret-name
EMQX Helm Chart 配置项
| 参数 | 描述 | Default Value |
|---|---|---|
replicaCount | EMQX 节点数量，建议保持奇数个节点，不然脑裂后无法自动恢复 | 3 |
image.repository | EMQX 镜像名称 | emqx/emqx |
image.pullPolicy | 获取镜像的策略 | IfNotPresent |
persistence.enabled | 是否启用 PVC | false |
persistence.storageClass | Storage class 名称 | nil |
persistence.existingClaim | PV 名称 | "" |
persistence.accessMode | PVC 访问模式 | ReadWriteOnce |
persistence.size | PVC 容量 | 20Mi |
resources | CPU/ 内存资源 | {} |
nodeSelector | pod 分配的节点标签 | {} |
tolerations | [] | |
affinity | {} | |
service.type | Emqx cluster service type | ClusterIP |
emqxConfig | EMQX 配置项，详情查看文档 | {} |
emqxLicneseSecretName | EMQX 企业版需要手动将 License 文件创建为 Secret 资源 (仅在 emqx/emqx-e 有效) | "" |
当需要设置复杂参数的时候，可以使用 Yaml 文件来记录参数
$ helm install my-emqx emqx/emqx -f values.yaml
你可以从 Github 获取默认的
values.yaml
地址：https://github.com/emqx/emqx-rel/tree/master/deploy/charts/emqx
二、通过原生yaml方式部署EMQX集群
首先，可以从方法一GitHub地址去获取默认的yaml文件，再通过修改想设置的参数去实现emqx集群部署。例如：
1.编写emqx.yaml
---
# Source: emqx-ee/templates/rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: emqx-ee
name: emqx-ee
---
# Source: emqx-ee/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: emqx-ee-env
namespace: emqx-ee
labels:
app.kubernetes.io/name: emqx-ee
data:
EMQX_CLUSTER__K8S__ADDRESS_TYPE: "hostname"
EMQX_CLUSTER__K8S__APISERVER: "https://kubernetes.default.svc:443"
EMQX_CLUSTER__K8S__SUFFIX: "svc.cluster.local"
EMQX_LOG__TO: "both"
---# Source: emqx-ee/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: emqx-ee-acl
namespace: emqx-ee
labels:
app.kubernetes.io/name: emqx-ee
data:
"acl.conf": >
{allow, {user, "dashboard"}, subscribe, ["$SYS/#"]}.
{allow, {ipaddr, "127.0.0.1"}, pubsub, ["$SYS/#", "#"]}.
{deny, all, subscribe, ["$SYS/#", {eq, "#"}]}.
---
# Source: emqx-ee/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: emqx-ee-loaded-plugins
namespace: emqx-ee
labels:
app.kubernetes.io/name: emqx-ee
data:
"loaded_plugins": |
{emqx_management, true}. {emqx_dashboard, true}. {emqx_rule_engine, true}. {emqx_modules, true}.
---
# Source: emqx-ee/templates/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: emqx-ee-loaded-modules
namespace: emqx-ee
labels:
app.kubernetes.io/name: emqx-ee
data:
"loaded_modules": |
[ { "name": "internal_acl", "enable": true, "configs": {"acl_rule_file": "etc/acl.conf"} }, { "name": "presence", "enable": true, "configs": {"qos": 0} }, { "name": "recon", "enable": true, "configs": {} }, { "name": "retainer", "enable": true, "configs": { "expiry_interval": 0, "max_payload_size": "1MB", "max_retained_messages": 0, "storage_type": "ram" } } ]
---
# Source: emqx-ee/templates/rbac.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: emqx-ee
name: emqx-ee
rules:
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
- watch
- list
---
# Source: emqx-ee/templates/rbac.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: emqx-ee
name: emqx-ee
subjects:
- kind: ServiceAccount
name: emqx-ee
namespace: emqx-ee
roleRef:
kind: Role
name: emqx-ee
apiGroup: rbac.authorization.k8s.io
---
# Source: emqx-ee/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: emqx-ee
namespace: emqx-ee
labels:
app.kubernetes.io/name: emqx-ee
spec:
type: NodePort
ports:
- name: mqtt
port: 1883
protocol: TCP
targetPort: mqtt
nodePort: null
- name: mqttssl
port: 8883
protocol: TCP
targetPort: mqttssl
nodePort: null
- name: mgmt
port: 8081
protocol: TCP
targetPort: mgmt
nodePort: null
- name: ws
port: 8083
protocol: TCP
targetPort: ws
nodePort: null
- name: wss
port: 8084
protocol: TCP
targetPort: wss
nodePort: null
- name: dashboard
port: 18083
protocol: TCP
targetPort: dashboard
nodePort: null
selector:
app.kubernetes.io/name: emqx-ee
---
# Source: emqx-ee/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
name: emqx-ee-headless
namespace: emqx-ee
labels:
app.kubernetes.io/name: emqx-ee
spec:
type: ClusterIP
publishNotReadyAddresses: true
sessionAffinity: None
clusterIP: None
ports:
- name: mqtt
port: 1883
protocol: TCP
targetPort: mqtt
- name: mqttssl
port: 8883
protocol: TCP
targetPort: mqttssl
- name: mgmt
port: 8081
protocol: TCP
targetPort: mgmt
- name: ws
port: 8083
protocol: TCP
targetPort: ws
- name: wss
port: 8084
protocol: TCP
targetPort: wss
- name: dashboard
port: 18083
protocol: TCP
targetPort: dashboard
- name: ekka
port: 4370
protocol: TCP
targetPort: ekka
selector:
app.kubernetes.io/name: emqx-ee
---
# Source: emqx-ee/templates/StatefulSet.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: emqx-ee
namespace: emqx-ee
labels:
app.kubernetes.io/name: emqx-ee
spec:
serviceName: emqx-ee-headless
podManagementPolicy: Parallel
updateStrategy:
type: RollingUpdate
replicas: 3
selector:
matchLabels:
app.kubernetes.io/name: emqx-ee
template:
metadata:
labels:
app: emqx-ee
version: 4.3.5
app.kubernetes.io/name: emqx-ee
spec:
volumes:
- name: emqx-loaded-plugins
configMap:
name: emqx-ee-loaded-plugins
items:
- key: loaded_plugins
path: loaded_plugins
- name: emqx-loaded-modules
configMap:
name: emqx-ee-loaded-modules
items:
- key: loaded_modules
path: loaded_modules
- name: emqx-acl
configMap:
name: emqx-ee-acl
items:
- key: acl.conf
path: acl.conf
- name: emqx-license
secret:
secretName: emqx-license
- name: host-time
hostPath:
path: /etc/localtime
serviceAccountName: emqx-ee
securityContext:
fsGroup: 1000
fsGroupChangePolicy: Always
runAsUser: 1000
supplementalGroups:
- 1000
containers:
- name: emqx
image: "emqx/emqx-ee:4.3.5"
imagePullPolicy: IfNotPresent
securityContext:
runAsNonRoot: true
runAsUser: 1000
ports:
- name: mqtt
containerPort: 1883
- name: mqttssl
containerPort: 8883
- name: mgmt
containerPort: 8081
- name: ws
containerPort: 8083
- name: wss
containerPort: 8084
- name: dashboard
containerPort: 18083
- name: ekka
containerPort: 4370
envFrom:
- configMapRef:
name: emqx-ee-env
env:
- name: EMQX_NAME
value: emqx
- name: EMQX_CLUSTER__K8S__APP_NAME
value: emqx
- name: EMQX_CLUSTER__DISCOVERY
value: k8s
- name: EMQX_CLUSTER__K8S__SERVICE_NAME
value: emqx-ee-headless
- name: EMQX_CLUSTER__K8S__NAMESPACE
value: emqx-ee
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 500m
memory: 512Mi
volumeMounts:
- name: emqx-acl
mountPath: "/opt/emqx/etc/acl.conf"
subPath: "acl.conf"
- name: emqx-loaded-plugins
mountPath: "/opt/emqx/data/loaded_plugins"
subPath: "loaded_plugins"
- name: emqx-loaded-modules
mountPath: "/opt/emqx/data/loaded_modules"
subPath: "loaded_modules"
- name: emqx-license
mountPath: "/opt/emqx/etc/emqx.lic"
subPath: "emqx.lic"
readOnly: true
- name: host-time
mountPath: /etc/localtime
readinessProbe:
httpGet:
path: /status
port: 8081
initialDelaySeconds: 5
periodSeconds: 52.创建namespace
kubectl create ns emqx-ee
3.创建secret（事先需要准备好相应的license文件）
kubectl create secret generic emqx-license --from-file=./emqx.lic -n emqx-ee
4.通过应用emqx.yaml文件创建emqx集群
kubectl apply -f emqx.yaml -n emqx-ee
5.在创建完成后，查看pod状态以及service
[root@chq ~]# kubectl get po -n emqx-ee NAME READY STATUS RESTARTS AGE emqx-ee-0 1/1 Running 0 22s emqx-ee-1 1/1 Running 0 22s emqx-ee-2 1/1 Running 0 22s [root@chq ~]# kubectl get svc -n emqx-ee NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE emqx-ee NodePort 10.233.50.73 <none> 1883:30674/TCP,8883:30855/TCP,8081:30793/TCP,8083:31601/TCP,8084:31692/TCP,18083:32238/TCP 29s emqx-ee-headless ClusterIP None <none> 1883/TCP,8883/TCP,8081/TCP,8083/TCP,8084/TCP,18083/TCP,4370/TCP 29s