AutoMQ to ClickHouse

BladePipe supports data replication from AutoMQ to ClickHouse. View supported migration, sync, verification, and connector capabilities.

Target DataSource：

ClickHouse

Connection

Function	Description
Incremental Data Sync	Allow subscribing to messages from the source Topic and transforming them into DML operations, namely, INSERT, UPDATE, DELETE.
Subscription Modification	Add, delete, or modify the subscribed topics. For more information, see Modify Subscription.
Position Resetting	Reset positions by timestamp to consume the data in a past period again.

Function	Description
Message Format	The following message formats are supported. See Message Format. CloudCanal format AlibabaCanal format
Write in Append Mode	INSERT and UPDATE statements are written in batches in append mode, and DELETE statements are executed individually through ALTER statements.
Scheduled Table Optimization	By setting the parameter autoOptimizeThresholdSec, the tables are optimized regularly.

Function

Description

Message Format

The following message formats are supported. See Message Format.

Write in Append Mode

INSERT and UPDATE statements are written in batches in append mode, and DELETE statements are executed individually through ALTER statements.

Scheduled Table Optimization

By setting the parameter autoOptimizeThresholdSec, the tables are optimized regularly.

Limit	Description
Creating Tables in the Target in Advance	Only support automatic Topic creation for messages.
Raw Message Format	Only support raw message replication from AutoMQ to AutoMQ, and Raw Message Format needs to be selected at both the Source and the Target.
Special Operation	Too many DELETE operations (>50 records/second) significantly affect data synchronization performance.
Target Table Engine	Only the following table engines and corresponding source table types are supported: MergeTree(tables without primary keys) ReplacingMergeTree(tables with primary keys) ReplicatedMergeTree(tables without primary keys) ReplicatedReplacingMergeTree(tables with primary keys)

Prerequisite	Description
Port Preparation	Allow the migration and sync node (Worker) to connect to the AutoMQ nodes.

Parameter	Description
schemaFormat	MQ Message format. For more information, see Message Format.
consumerGroupId	AutoMQ consumer group ID.
consumeParallel	Degree of consuming AutoMQ topics in parallel.
sessionTimeoutMs	AutoMQ session timeout in milliseconds.
maxPollRecords	Maximum number of messages fetched in one poll from AutoMQ.

Tips: To modify the general parameters, see General Parameters and Functions.

Prerequisite	Description
Permissions for Account	SELECT, INSERT and common DDL permissions.
Port Preparation	Allow the migration and sync node (Worker) to connect to the ClickHouse port (e.g., 8123).

Parameter	Description
multiReplica	Whether there are multiple replicas in a cluster.
clusterName	Cluster name. When multiReplica is true, the ON CLUSTER clusterName clause is automatically added to DDL/DML.
ckTableEngine	The following table engines are currently supported: MergeTree ReplacingMergeTree ReplicatedMergeTree ReplicatedReplacingMergeTree
autoOptimizeThresholdSec	Interval of scheduled table optimization (optimize table final). If the value <=0, it means the feature is disabled.
enableTimeRangeClamping	Whether to enable time range clamping. Forces date and time values to be constrained within the valid ClickHouse JDBC range. Values outside this range will be clamped to the minimum or maximum values. Disabled by default (false). Ranges after clamping(UTC)： Date：[1970-01-01, 2149-06-06] Date32：[1925-01-01, 2283-11-11] Timestamp：[1970-01-01 00:00:00, 2106-02-07 14:28:15] Timestamp64：[1925-01-01 08:00:00.000, 2283-11-12 07:59:59.000]

Tips: To modify the general parameters, see General Parameters and Functions.