SQL Server 到 Iceberg

选择对端数据库：

Iceberg

数据链路

基本功能

功能	说明
Schema Migration	If the target schema does not exist, BladePipe will automatically generate and execute CREATE statements based on the source metadata and the mapping rule.
Full Data Migration	Migrate data by sequentially scanning data in tables and writing it in batches to the target database.
Incremental Data Sync	Sync of common DML like INSERT, UPDATE, DELETE is supported. UPDATE and DELETE for tables without primary keys are not synced by default (manual selection required).
Subscription Modification	Add, delete, or modify the subscribed tables with support for historical data migration. For more information, see Modify Subscription.
Table Name Mapping	Support the mapping rules, namely, keeping the name the same as that in Source, converting the text to lowercase, converting the text to uppercase, truncating the name by "_digit" suffix.
DDL Sync	Supports ALTER TABLE ADD COLUMN and DROP COLUMN.

高级功能

功能	说明
写入冲突策略	源端有主键表进行覆盖写入，源端无主键表进行追加写入
自定义表属性	包括 format-version 等属性设置
设置数据分区	创建任务时，可按表粒度指定分区定义（静态或动态），结构迁移时自动添加该分区定义
Custom Code	For more information, see Custom Code Processing, Debug Custom Code and Logging in Custom Code.
Setting Target Primary Key	Change the primary key to another field to facilitate data aggregation and other operations.
Data Filtering Conditions	Support data filtering using WHERE conditions, with SQL-92 as the SQL language. For more information, see Data Filtering.

链路FAQ

• 出现 TLS10 is not accepted 问题该怎么办?

---

源端数据源

前置条件

条件	说明
账号权限	文档：SQL Server 需要的权限
开启 SQL Server CDC	exec [你的数据库].sys.sp_cdc_enable_db

任务参数

参数名称	说明
maxTxsPerIteration	SQL Server 增量源端 CDC 每次最大扫描事务数
scanParallel	全量阶段：并发扫描表数量 增量阶段：并发扫描 CDC 表数量
eventStoreSize	缓存解析完毕的增量事件缓存大小

Tips: 通用参数配置请参考 通用参数及功能

---

目标端数据源

前置条件

条件	说明
网络准备	迁移同步节点（sidecar）可连接 Catalog 和 文件存储
Nessie 数据源配置模版	网络地址（CatalogUri）: ip:19120/api/v1 catalogName: nessie catalogType: NESSIE catalogWarehouse: s3://warehouse catalogProps: { "io-impl": "org.apache.iceberg.aws.s3.S3FileIO", "s3.endpoint": "http://ip:9000", "s3.access-key-id": "admin", "s3.secret-access-key": "password", "s3.path-style-access": "true", "client.region": "ap-southeast-1" }
Glue 数据源配置模版	网络地址（CatalogUri）: glue.ap-southeast-1.amazonaws.com httpsEnabled: true catalogName: glue_catalog catalogType: GLUE catalogWarehouse : s3://warehouse catalogProps: { "io-impl": "org.apache.iceberg.aws.s3.S3FileIO", "s3.endpoint": "https://s3.ap-southeast-1.amazonaws.com", "s3.access-key-id": "", "s3.secret-access-key": "", "s3.path-style-access": "true", "client.region": "ap-southeast-1", "client.credentials-provider.glue.access-key-id": "", "client.credentials-provider.glue.secret-access-key": "", "client.credentials-provider": "com.amazonaws.glue.catalog.credentials.GlueAwsCredentialsProvider" }
REST 数据源配置模版	网络地址（CatalogUri）: ip :8181 httpsEnabled: false catalogName: rest_catalog catalogType: REST catalogWarehouse : s3://warehouse catalogProps: { "io-impl": "org.apache.iceberg.aws.s3.S3FileIO", "s3.endpoint": "http://ip:9000", "s3.access-key-id": "admin", "s3.secret-access-key": "password", "s3.path-style-access": "true", "client.region": "us-east-1" }

任务参数

参数名称	说明
fileFormat	写入文件格式（parquet / orc / ... ）
writeTargetFileSizeMb	写入目标文件大小（MB）
writeProps	写入配置参数（Json 格式）
commitBranch	写入提交的分支
totalDataInMemMb	攒批写入，内存中最大数据容量，超过此容量或超过 asyncFlushIntervalSec 则刷出数据到写入队列
asyncFlushIntervalSec	攒批写入，等待刷出的间隔时间，超过此时间或超过 totalDataInMemMb 则刷出数据到写入队列
flushBatchMb	单表最大攒批容量，超过此容量则刷出数据到写入队列
realFlushPauseSec	刷出数据到 Iceberg 的等待时间，0 则不等待

Tips: 通用参数配置请参考 通用参数及功能