ClickHouse集群部署（一）

EMQX 客服 • 发表于：2022年03月04日 16:41:31 • 更新于：2022年03月07日 14:16:29

一、高可用架构

image.png

ClickHouse集群：

3个节点：1个节点,2个ClickHouse实例；3分片,2副本：

Centos-1: 实例1, 端口: tcp 9000, http 8123, 同步端口9009, 类型: 分片1, 副本1

Centos-1: 实例2, 端口: tcp 9002, http 8124, 同步端口9010, 类型: 分片3, 副本2 (Centos-2的副本)

Centos-2: 实例1, 端口: tcp 9000, http 8123, 同步端口9009, 类型: 分片2, 副本1

Centos-2: 实例2, 端口: tcp 9002, http 8124, 同步端口9010, 类型: 分片1, 副本2 (Centos-3的副本)

Centos-3: 实例1, 端口: tcp 9000, http 8123, 同步端口9009, 类型: 分片3, 副本1

Centos-3: 实例2, 端口: tcp 9002, http 8124, 同步端口9010, 类型: 分片2, 副本2 (Centos-1的副本)

二、环境准备

  1. 软件

    名称版本备注
    操作系统Centos7.8
    zookeeper3.7.0
    clickhouse21.12.3
    jdk1.8.0_161

  2. 硬件

    IPhostsoftware
    192.168.0.27
    		ecs-clickhouse-001
    		jdk,zookeeper,clickhouse
    192.168.0.221
    		ecs-clickhouse-002
    		jdk,zookeeper,clickhouse
    192.168.0.74
    		ecs-clickhouse-003
    		jdk,zookeeper,clickhouse

三、部署

  1. JDK安装

    在3台Linux裸机服务器上分别安装JDK(要求版本不低于JDK8)

    （1）官网或者其他途径下载jdk 安装包

    （2）解压安装包

    tar -xvf jdk-8u161-linux-x64.tar.gz

    image.png

    （3）将解压包拷贝到/usr/java/下

    mkdir -p /usr/java
cp -r jdk1.8.0_161 /usr/java/

    （4）添加环境变量

    在/etc/profile文件最后添加变量

    export JAVA_HOME=/usr/java/jdk1.8.0_161
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib:$CLASSPATH
export JAVA_PATH=${JAVA_HOME}/bin:${JRE_HOME}/bin
export PATH=$PATH:${JAVA_PATH}

    image.png

    （5）设置配置文件生效

    source /etc/profile
java -version

    image.png

  2. zookeeper集群安装

    （1）下载安装包

    wget https://dlcdn.apache.org/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz

    image.png

    （2）解压到/usr/local

    tar -zxvf apache-zookeeper-3.7.0-bin.tar.gz -C /usr/local/
cd /usr/local
mv apache-zookeeper-3.7.0-bin zookeeper

    image.png

    （3）配置文件

    进入到/usr/local/zookeeper/conf/下，拷贝文件zoo_sample.cfg 为zoo.cfg

    cp zoo_sample.cfg zoo.cfg

    image.png

    修改文件zoo.cfg

    image.png

    initLimit

    tickTime的个数，表示在leader选举结束后，followers与leader同步需要的时间，如果followers比较多或者说leader的数据灰常多时，同步时间相应可能会增加，那么这个值也需要相应增加。当然，这个值也是follower和observer在开始同步leader的数据时的最大等待时间(setSoTimeout)

    syncLimit

    tickTime的个数，这时间容易和上面的时间混淆，它也表示follower和observer与leader交互时的最大等待时间，只不过是在与leader同步完毕之后，进入正常请求转发或ping等消息交互时的超时时间。

    leaderServes

    (Java system property: zookeeper.leaderServes)  如果该值不是no，则表示该服务器作为leader时是需要接受客户端连接的。为了获得更高吞吐量，当服务器数三台以上时一般建议设置为no。

    cnxTimeout

    (Java system property: zookeeper.cnxTimeout) 默认值是5000,单位ms 表示leaderelection时打开连接的超时时间，只用在算法3中。

    preAllocSize

    (Java system property: zookeeper.preAllocSize)默认值64M，以KB为单位,预先分配额定空间用于后续transactionlog 写入，每当剩余空间小于4K时，就会又分配64M，如此循环。如果SNAP做得比较频繁(snapCount比较小的时候)，那么请减少这个值。

    snapCount

    (Java system property: zookeeper.snapCount)默认值100,000，当transaction每达到snapCount/2+rand.nextInt(snapCount/2)时，就做一次SNAPSHOT,默认情况下是50,000~100,000条transactionlog就会做一次，之所以用随机数是为了避免所有服务器可能在同一时间做snapshot.

    maxClientCnxns

    默认值是10，一个客户端能够连接到同一个服务器上的最大连接数，根据IP来区分。如果设置为0，表示没有任何限制。设置该值一方面是为了防止DoS攻击。

    maxSessionTimeout

    最大的session time 时间，默认值是20个tick time. ,客户端设置的session time 如果大于这个值，则会被强制协调为这个最大值。

    （4）拷贝到其他几台zookeeper机器上

    从第一台拷贝到第二台：scp -r zookeeper root@192.168.0.221:/usr/local/

    image.png

    查看第二台结果：

    image.png

    从第一台拷贝到第三台同样方式

    （5）创建目录（zookeeper每台机器执行）

    mkdir -p /data/zookeeper/{data,logs}

    image.png

    （6）创建myid文件

    ecs-clickhouse-001

    echo 1 > /data/zookeeper/data/myid

    image.png

    ecs-clickhouse-002

    echo 2 > /data/zookeeper/data/myid

    image.png

    ecs-clickhouse-003

    echo 3 > /data/zookeeper/data/myid

    image.png

    （7）配置环境变量并生效（zookeeper每台都需执行）

    在/etc/profile 文件最后添加变量

    export ZOOKEEPER_HOME=/usr/local/zookeeper

    export PATH=$PATH:$ZOOKEEPER_HOME/bin

    image.png

    环境变量生效

    source /etc/profile

    （8）配置zookeeper日志

    默认zk日志输出到一个文件,且不会自动清理,所以,一段时间后zk日志会非常大。

     zookeeper-env.sh ./conf目录下新建zookeeper-env.sh文件,修改到sudo chmod 755 zookeeper-env.sh权限

    先创建目录：/usr/local/zookeeper/logs

    image.png

    log4j.properties修改日志的输入形式

    image.png

    （9）配置运行zk的JVM

    conf目录下新建java.env文件,修改到sudo chmod 755 java.env权限,主要用于GC log,RAM等的配置.

    image.png

    （10）启动

    3台zookeeper都需要启动

    /usr/local/zookeeper/bin/zkServer.sh start

    image.png

    （11）检查服务是否正常

    3台zookeeper 分别执行，1个leader ，2个follower

    /usr/local/zookeeper/bin/zkServer.sh status

    image.png

    （12）验证zk

    ./bin/zkCli.sh -server 192.168.0.27:2181

    image.png

  3. clickhouse集群安装

    （1）检查CPU支持SSE4.2指令集

    grep -q sse4_2 /proc/cpuinfo && echo “SSE 4.2 supported” || echo “SSE 4.2 not supported”

    image.png

    （2）查看Linux版本

    cat  /proc/version

    image.png

    (3)根据Linux版本（el7.x86_64）下载对应的安装包

    安装包下载源配置

    yum install yum-utils

    rpm --import https://repo.clickhouse.tech/CLICKHOUSE-KEY.GPG

    yum-config-manager --add-repo https://repo.clickhouse.tech/rpm/stable/x86_64

    image.png

    离线下载安装包，放到/root/clickhouse/下

    yum install -y clickhouse-server clickhouse-client --downloadonly --downloaddir=/root/clickhouse/

    image.png

    （4）安装（将安装包拷贝到其他几台机器上，分别执行）

    拷贝到其他几台机器上：

    image.png

    进入安装包目录/root/clickhouse/,执行安装命令：

    rpm -ivh clickhouse-common-static-21.12.3.32-2.x86_64.rpm clickhouse-server-21.12.3.32-2.noarch.rpm clickhouse-client-21.12.3.32-2.noarch.rpm

    中间需要输入默认密码，回车就行，不需要输入，等后期在配置。

    image.png

    image.png

    （5）创建目录

    ecs-clickhouse-001:

    mkdir -p /data/clickhouse/{node1,node4}/{data,tmp,logs}

    image.png

    ecs-clickhouse-002:

    mkdir -p /data/clickhouse/{node2,node5}/{data,tmp,logs}

    image.png

    ecs-clickhouse-003:

    mkdir -p /data/clickhouse/{node3,node6}/{data,tmp,logs}

    image.png

    （6）用户名密码生成

    超级用户:default

    密码：supersecrect

    密码password_sha256_hex：echo -n "supersecrect" | sha256sum | tr -d '-'

    image.png

    普通用户:emqx

    密码：emqx

    密码password_sha256_hex：echo -n "emqx" | sha256sum | tr -d '-'

    image.png

    （7）创建配置文件

    node1配置文件：

    config.xml
    <?xml version="1.0"?>
    <yandex>    
        <!--日志-->    
        <logger>        
            <level>warning</level>        
            <log>/data/clickhouse/node1/logs/clickhouse.log</log>        
            <errorlog>/data/clickhouse/node1/logs/error.log</errorlog>        
            <size>500M</size>        
            <count>5</count>    
        </logger>    
        <!--本地节点信息-->    
        <http_port>8123</http_port>    
        <tcp_port>9000</tcp_port>    
        <interserver_http_port>9009</interserver_http_port>    
        <interserver_http_host>192.168.0.27</interserver_http_host>    
        <!--本机域名或IP-->    
        <!--本地配置-->    
        <listen_host>0.0.0.0</listen_host>    
        <max_connections>2048</max_connections>    
        <keep_alive_timeout>3</keep_alive_timeout>    
        <max_concurrent_queries>64</max_concurrent_queries>    
        <uncompressed_cache_size>4294967296</uncompressed_cache_size>    
        <mark_cache_size>5368709120</mark_cache_size>    
        <path>/data/clickhouse/node1/</path>    
        <tmp_path>/data/clickhouse/node1/tmp/</tmp_path>    
        <users_config>/data/clickhouse/node1/users.xml</users_config>    
        <default_profile>default</default_profile>    
        <query_log>        
            <database>system</database>        
            <table>query_log</table>        
            <partition_by>toMonday(event_date)</partition_by>        
            <flush_interval_milliseconds>7500</flush_interval_milliseconds>    
        </query_log>    
        <query_thread_log>        
            <database>system</database>        
            <table>query_thread_log</table>        
            <partition_by>toMonday(event_date)</partition_by>        
            <flush_interval_milliseconds>7500</flush_interval_milliseconds>    
        </query_thread_log>    
        <prometheus>        
            <endpoint>/metrics</endpoint>        
            <port>8001</port>        
            <metrics>true</metrics>        
            <events>true</events>        
            <asynchronous_metrics>true</asynchronous_metrics>    
        </prometheus>    
        <default_database>default</default_database>    
        <timezone>Asia/Shanghai</timezone>    
        <!--集群相关配置-->    
        <remote_servers incl="clickhouse_remote_servers" />    
        <zookeeper incl="zookeeper-servers" optional="true" />    
        <macros incl="macros" optional="true" />    
    <builtin_dictionaries_reload_interval>3600</builtin_dictionaries_reload_interval>    
        <max_session_timeout>3600</max_session_timeout>    
        <default_session_timeout>300</default_session_timeout>    
        <max_table_size_to_drop>0</max_table_size_to_drop>    
        <merge_tree>        
            <parts_to_delay_insert>300</parts_to_delay_insert>        
            <parts_to_throw_insert>600</parts_to_throw_insert>        
            <max_delay_to_insert>2</max_delay_to_insert>    
        </merge_tree>    
        <max_table_size_to_drop>0</max_table_size_to_drop>    
        <max_partition_size_to_drop>0</max_partition_size_to_drop>    
        <distributed_ddl>        
            <!-- Path in ZooKeeper to queue with DDL queries -->        
            <path>/clickhouse/task_queue/ddl</path>    
        </distributed_ddl>    
        <include_from>/data/clickhouse/node1/metrika.xml</include_from>
</yandex>

    image.png

    image.png

    metrika.xml
    <?xml version="1.0"?>
    <yandex>    
        <!--ck集群节点-->    
        <clickhouse_remote_servers>        
            <emqx_cluster_all>            
            <!--分片1-->            
            <shard>                
                <internal_replication>true</internal_replication>                
                <replica>                    
                    <host>192.168.0.27</host>                    
                    <port>9000</port>                    
                    <user>default</user>                    
                    <password>supersecrect</password>                
                </replica>                
                <!--复制集1-->                
                <replica>                    
                    <host>192.168.0.74</host>                    
                    <port>9002</port>                    
                    <user>default</user>                    
                    <password>supersecrect</password>                
                </replica>            
            </shard>            
            <!--分片2-->            
            <shard>                
                <internal_replication>true</internal_replication>                
                <replica>                    
                    <host>192.168.0.221</host>                    
                    <port>9000</port>                    
                    <user>default</user>                    
                    <password>supersecrect</password>                
                </replica>                
                <!--复制集2-->                
                <replica>                    
                    <host>192.168.0.27</host>                    
                    <port>9002</port>                    
                    <user>default</user>                    
                    <password>supersecrect</password>                
                </replica>            
            </shard>            
            <!--分片3-->            
            <shard>                
            <internal_replication>true</internal_replication>                
            <replica>                    
                <host>192.168.0.74</host>                    
                <port>9000</port>                    
                <user>default</user>                    
                <password>supersecrect</password>                
            </replica>                
            <!--复制集3-->                
            <replica>                    
                <host>192.168.0.221</host>                    
                <port>9002</port>                    
                <user>default</user>                    
                <password>supersecrect</password>                
            </replica>            
        </shard>        
    </emqx_cluster_all>    
</clickhouse_remote_servers>    
<!--zookeeper相关配置-->    
<zookeeper-servers>        
    <node index="1">            
        <host>192.168.0.27</host>            
        <port>2181</port>        
    </node>        
    <node index="2">            
        <host>192.168.0.221</host>            
        <port>2181</port>        
    </node>        
    <node index="3">            
        <host>192.168.0.74</host>            
        <port>2181</port>        
    </node>    
</zookeeper-servers>    
<macros>        
    <layer>01</layer>        
    <shard>01</shard>        
    <!--分片号-->        
    <replica>node1</replica>        
    <!--当前节点IP-->    
</macros>    
<networks>        
    <ip>::/0</ip>    
</networks>    
<!--压缩相关配置-->    
<clickhouse_compression>        
    <case>            
        <min_part_size>10000000000</min_part_size>            
        <min_part_size_ratio>0.01</min_part_size_ratio>            
        <method>lz4</method>            
        <!--压缩算法lz4压缩比zstd快, 更占磁盘-->        
    </case>    
</clickhouse_compression>
</yandex>

    image.png

    image.png

    users.xml
    <?xml version="1.0"?>
    <yandex>    
        <profiles>        
            <default>            
                <!-- 请根据自己机器实际内存配置 -->            
                <max_memory_usage>54975581388</max_memory_usage>            
                <max_memory_usage_for_all_queries>61847529062</max_memory_usage_for_all_queries>            
                <max_bytes_before_external_group_by>21474836480</max_bytes_before_external_group_by>            
                <max_bytes_before_external_sort>21474836480</max_bytes_before_external_sort>            
                <use_uncompressed_cache>0</use_uncompressed_cache>            
                <load_balancing>random</load_balancing>            
                <distributed_aggregation_memory_efficient>1</distributed_aggregation_memory_efficient>           
                <max_threads>8</max_threads>            
                <log_queries>1</log_queries>            
                <receive_timeout>800</receive_timeout>            
                <send_timeout>800</send_timeout>        
        </default>        
        <readonly>            
            <max_threads>8</max_threads>            
            <max_memory_usage>54975581388</max_memory_usage>            
            <max_memory_usage_for_all_queries>61847529062</max_memory_usage_for_all_queries>            
            <max_bytes_before_external_group_by>21474836480</max_bytes_before_external_group_by>            
            <max_bytes_before_external_sort>21474836480</max_bytes_before_external_sort>            
            <use_uncompressed_cache>0</use_uncompressed_cache>            
            <load_balancing>random</load_balancing>            
            <readonly>1</readonly>            
            <distributed_aggregation_memory_efficient>1</distributed_aggregation_memory_efficient>            
            <log_queries>1</log_queries>        
        </readonly>    
    </profiles>    
    <quotas>        
        <default>            
            <interval>                
                <duration>3600</duration>                
                <queries>0</queries>                
                <errors>0</errors>                
                <result_rows>0</result_rows>                
                <read_rows>0</read_rows>                
                <execution_time>0</execution_time>            
            </interval>        
    </default>    
    </quotas>    
    <users>        
        <default>            
            <password>supersecrect</password>            
                <networks>                
                    <ip>::/0</ip>            
                </networks>            
                <profile>default</profile>            
                <quota>default</quota>        
        </default>        
        <emqx>            
        <password_sha256_hex>fe56917863d5980019f4b6405239aa2d8e8a967b68c30987dc65f60eaf56cec3</password_sha256_hex>            
            <networks>                
                <ip>::/0</ip>            
            </networks>            
            <profile>default</profile>            
            <quota>default</quota>        
        </emqx>        
        <read>            
        <password_sha256_hex>fe56917863d5980019f4b6405239aa2d8e8a967b68c30987dc65f60eaf56cec3</password_sha256_hex>            
            <networks>                
                <ip>::/0</ip>            
            </networks>            
            <profile>readonly</profile>            
            <quota>default</quota>        
        </read>    
    </users>
</yandex>

    image.png

    node2配置文件：

    config.xml
    <?xml version="1.0"?>
    <yandex>    
        <!--日志-->    
        <logger>        
            <level>warning</level>        
            <log>/data/clickhouse/node2/logs/clickhouse.log</log>        
            <errorlog>/data/clickhouse/node2/logs/error.log</errorlog>        
            <size>500M</size>        
            <count>5</count>    
        </logger>    
        <!--本地节点信息-->    
        <http_port>8123</http_port>    
        <tcp_port>9000</tcp_port>    
        <interserver_http_port>9009</interserver_http_port>    
        <interserver_http_host>192.168.0.221</interserver_http_host>    
        <!--本机域名或IP-->    
        <!--本地配置-->    
        <listen_host>0.0.0.0</listen_host>    
        <max_connections>2048</max_connections>    
        <keep_alive_timeout>3</keep_alive_timeout>    
        <max_concurrent_queries>64</max_concurrent_queries>    
        <uncompressed_cache_size>4294967296</uncompressed_cache_size>    
        <mark_cache_size>5368709120</mark_cache_size>    
        <path>/data/clickhouse/node2/</path>    
        <tmp_path>/data/clickhouse/node2/tmp/</tmp_path>    
        <users_config>/data/clickhouse/node2/users.xml</users_config>    
        <default_profile>default</default_profile>    
        <query_log>        
            <database>system</database>        
            <table>query_log</table>        
            <partition_by>toMonday(event_date)</partition_by>        
            <flush_interval_milliseconds>7500</flush_interval_milliseconds>    
        </query_log>    
        <query_thread_log>        
            <database>system</database>        
            <table>query_thread_log</table>        
            <partition_by>toMonday(event_date)</partition_by>        
            <flush_interval_milliseconds>7500</flush_interval_milliseconds>    
        </query_thread_log>    
        <prometheus>        
            <endpoint>/metrics</endpoint>        
            <port>8001</port>        
            <metrics>true</metrics>        
            <events>true</events>        
            <asynchronous_metrics>true</asynchronous_metrics>    
        </prometheus>    
        <default_database>default</default_database>    
        <timezone>Asia/Shanghai</timezone>    
        <!--集群相关配置-->    
        <remote_servers incl="clickhouse_remote_servers" />    
        <zookeeper incl="zookeeper-servers" optional="true" />    
        <macros incl="macros" optional="true" />    
        <builtin_dictionaries_reload_interval>3600</builtin_dictionaries_reload_interval>    
        <max_session_timeout>3600</max_session_timeout>    
        <default_session_timeout>300</default_session_timeout>    
        <max_table_size_to_drop>0</max_table_size_to_drop>    
        <merge_tree>        
            <parts_to_delay_insert>300</parts_to_delay_insert>        
            <parts_to_throw_insert>600</parts_to_throw_insert>        
            <max_delay_to_insert>2</max_delay_to_insert>    
        </merge_tree>    <max_table_size_to_drop>0</max_table_size_to_drop>    
        <max_partition_size_to_drop>0</max_partition_size_to_drop>    
        <distributed_ddl>        
            <!-- Path in ZooKeeper to queue with DDL queries -->        
            <path>/clickhouse/task_queue/ddl</path>    
        </distributed_ddl>    
        <include_from>/data/clickhouse/node2/metrika.xml</include_from>
    </yandex>

    image.png

    image.png


    您需要登录后才可以回复

    最多浏览的文档