Oracle to Kafka
BladePipe supports data replication from Oracle to Kafka. View supported migration, sync, verification, and connector capabilities.
| Function | Description |
|---|---|
Full Data Migration | Migrate data by sequentially scanning data in tables and writing it in batches to the message-oriented middleware. |
Incremental Data Sync | Sync of common DML like INSERT, UPDATE, DELETE is supported. |
Subscription Modification | Add, delete, or modify the subscribed tables with support for historical data migration. For more information, see Modify Subscription. |
Position Resetting | Reset the position by timestamp or Scn to consume Oracle Redo Log in a past period again. |
Metadata Retrieval | Retrieve the target metadata with filtering conditions from the source table. |
Advanced Functions
| Function | Description |
|---|---|
Automatic Dictionary Creation | When using offline dictionaries to parse Oracle Redo files, the dictionary is automatically created during DataJob creation. |
Message Format | The following message formats are supported. See Message Format.
|
Topic Mapping Rules | By default, Topic is formed by connecting source instance id, database, and table with . in between (e.g., ora-vgpq6q097174t6t.ORCL.dingtax.app_key). Also, it supports the mapping rules, namely, keeping the name the same as that in Source, converting the text to lowercase, converting the text to uppercase. |
Table-level Topic | Create Topics corresponding to the tables in the Source, and the table partitions can be obtained automatically. |
DDL Dedicated Topic | Allow specifying a Topic for DDL. If not specified, DDL time is placed in partition 0 of the Topic created from the corresponding table. |
Scheduled Full Data Migration | For more information, see Create Scheduled Full Data DataJob. |
Custom Code | For more information, see Custom Code Processing, Debug Custom Code and Logging in Custom Code. |
Data Filtering Conditions | Support data filtering using WHERE conditions, with SQL-92 as the SQL language. For more information, see Data Filtering. |
Setting Target Primary Key | Change the primary key to another field. |
Limits
| Limit | Description |
|---|---|
Incremental Data Sync Performance | Due to LogMiner performance limits and BladePipe's lack of parallel analysis, the performance benchmark is set at 3000 records per second. |
Data Types | Do not support BLOB and derived types. |
Prerequisites
| Prerequisite | Description |
|---|---|
Permissions for Account | |
Incremental Data Sync Preparation | |
Port Preparation | Allow the migration and sync node (Worker) to connect to the Oracle port (e.g., 1521). |
Parameters
| Parameter | Description |
|---|---|
fullFetchSize | Fetch size for scaning full data. |
eventStoreSize | Cache size for parsed incremental events. |
logminerUser | User name for connection to Oracle to execute LogMiner SQL. |
logminerPasswd | Password for connection to Oracle to execute LogMiner SQL. |
logminerConnectType | Way to connect to Oracle (PDB) to execute LogMiner SQL, including ORACLE_SID and ORACLE_SERVICE options. |
logminerSidOrService | SID or service name for connection to Oracle (PDB) to execute LogMiner SQL. |
parseRedoSqlParallel | Number of threads for parallel parsing of LogMiner data. |
parseRedoSqlBufferSize | Size of the circular buffer for parsing LogMiner data. |
redoFetchSize | Number of rows of LogMiner analyzed data to fetch each time. |
redoOfferTransMaxSize | Maximum number of unconsumed but committed transactions in cache. |
oraMiningSessionPauseSec | Interval between LogMiner sessions, in seconds. |
maxEventCountPerTxInMem | Maximum number of events per transaction in memory. |
logMiningScnStep | Analysis range specified when Oracle LogMiner analyzes redo logs. |
abandonUnCommitTxTimeoutSec | Automatically abandon transactions that have not been committed for longer than the specified timeout. |
restartTxWithDataTimeoutSec | Automatically restart DataJobs for transactions with data changes that have not been committed for longer than the specified timeout. |
oraUseOnlineDic | Whether to use online logs; false means using offline logs, which may put more stress on Oracle. |
oraReleaseIntervalSec | Interval for recreating connection for analysis to release Oracle server resources. |
oraMiningSessionPauseSec | Interval between execution of LogMiner commands for analysis. |
fallBackScnStep | Distance to keep from the latest Redo log data; 0 means to keep right behind the latest Redo log data. |
sqlCaseConversionEnabled | Whether to enable DDL case conversion (according to the default case rules of the current database). |
Tips: To modify the general parameters, see General Parameters and Functions.
Prerequisites
| Prerequisite | Description |
|---|---|
Port Preparation | Allow the migration and sync node (Worker) to connect to the Kafka ports. |
Parameters
| Parameter | Description |
|---|---|
schemaFormat | Message format. For more information, see Message Format. |
batchWriteSize | The maximum data size of a single message. If the size exceeds the limit, the message will be split. |
defaultTopic | Messages that cannot find a corresponding topic are sent to this topic (such as adding a new table) |
ddlTopic | A topic specifically used to receive DDL events. If it is empty, the DDL events will be sent to the 0th partition of the corresponding topic. |
compressionType | Kafka compression.type parameter to set compression algorithm. Support GZIP, SNAPPY, LZ4, ZSTD. |
batchSize | Kafka batch.size parameter. |
acks | Kafka acks parameter. By default, it is all. |
maxRequestBytes | Kafka max.request.size parameter. |
lingerMs | Kafka linger.ms parameter. By default, it is 1. |
envelopSchemaInclude | When schemaFormat is set to DEBEZIUM_ENVELOP_JSON_FOR_MQ, it means whether the message body contains schema information. |
customClientProps | Custom properties passed to the Kafka client in JSON format. The key is the parameter name and the value is the parameter value. This setting takes the highest priority. For example: AWS IAM Access Control |
Tips: To modify the general parameters, see General Parameters and Functions.