Oracle to Elasticsearch
BladePipe supports data replication from Oracle to Elasticsearch. View supported migration, sync, verification, and connector capabilities.
| Function | Description |
|---|---|
Schema Migration | If the target index does not exist, BladePipe will create the index mapping in the target based on the source metadata and mapping rules. |
Full Data Migration | Migrate data by sequentially scanning data in tables and writing it in batches to the target database. |
Incremental Data Sync | Sync of common DML like INSERT, UPDATE, DELETE is supported. |
Data Verification and Correction | Verify all existing data. Optionally, you can correct the inconsistent data based on verification results. Scheduled DataTasks are supported. |
Subscription Modification | Add, delete, or modify the subscribed tables with support for historical data migration. For more information, see Modify Subscription. |
Position Resetting | Reset the position by timestamp or Scn to consume Oracle Redo Log in a past period again. |
Index Name Mapping | Support the mapping rules, namely, concatenation with underscores (dataJobName_DB_SCHEMA_table), keeping the name the same as that in Source, converting the text to lowercase, converting the text to uppercase, truncating the name by "_digit" suffix. |
DDL Synchronization |
|
Metadata Retrieval | Retrieve the target metadata with filtering conditions or target primary keys set from the source table. |
Advanced Functions
| Function | Description |
|---|---|
Automatic Dictionary Creation | When using offline dictionaries to parse Oracle Redo files, the dictionary is automatically created during DataJob creation. |
Removal of Target Data before Full Data Migration | Remove the existing data in the Target before running the Full Data Migration, applicable for DataJobs reruning and scheduled Full Data migrations. |
Recreating Target Table | Recreate target tables before running the Full Data Migration, applicable for DataJobs reruning and scheduled Full Data migrations. |
Format of Time Written to ES | Time is written to Elasticsearch in the format of the first time record of the field, or yyyy-MM-dd'T'HH:mm:ss if no time format is set. |
Setting ES Time Zone | The time zone setting on the page will be written to Elasticsearch only when the time zone format is ZZZZZ. |
Optional Fields in Indexing | By default, all fields are indexed. Specific fields can be excluded from indexing. |
Field-level Analyzers | Allow selecting analyzers for string fields that are indexed. Support STANDARD (default), SIMPLE, and other common analyzers, with the option to specify custom analyzers. |
Setting Index _id Field | By default, the _id field is a concatenation of the source primary key values. It can be changed to other field values. |
Scheduled Full Data Migration | For more information, see Create Scheduled Full Data DataJob. |
Custom Code | For more information, see Custom Code Processing, Debug Custom Code and Logging in Custom Code. |
Data Filtering Conditions | Support data filtering using WHERE conditions, with SQL-92 as the SQL language. For more information, see Data Filtering. |
Setting Target Primary Key | Change the primary key to another field to facilitate data aggregation and other operations. |
Limits
| Limit | Description |
|---|---|
Incremental Data Sync Performance | Due to LogMiner performance limits and BladePipe's lack of parallel analysis, the performance benchmark is set at 3000 records per second. |
Data Types | Do not support BLOB and derived types. |
Prerequisites
| Prerequisite | Description |
|---|---|
Permissions for Account | |
Incremental Data Sync Preparation | |
Port Preparation | Allow the migration and sync node (Worker) to connect to the Oracle port (e.g., 1521). |
Parameters
| Parameter | Description |
|---|---|
fullFetchSize | Fetch size for scaning full data. |
eventStoreSize | Cache size for parsed incremental events. |
logminerUser | User name for connection to Oracle to execute LogMiner SQL. |
logminerPasswd | Password for connection to Oracle to execute LogMiner SQL. |
logminerConnectType | Way to connect to Oracle (PDB) to execute LogMiner SQL, including ORACLE_SID and ORACLE_SERVICE options. |
logminerSidOrService | SID or service name for connection to Oracle (PDB) to execute LogMiner SQL. |
parseRedoSqlParallel | Number of threads for parallel parsing of LogMiner data. |
parseRedoSqlBufferSize | Size of the circular buffer for parsing LogMiner data. |
redoFetchSize | Number of rows of LogMiner analyzed data to fetch each time. |
redoOfferTransMaxSize | Maximum number of unconsumed but committed transactions in cache. |
oraMiningSessionPauseSec | Interval between LogMiner sessions, in seconds. |
maxEventCountPerTxInMem | Maximum number of events per transaction in memory. |
logMiningScnStep | Analysis range specified when Oracle LogMiner analyzes redo logs. |
abandonUnCommitTxTimeoutSec | Automatically abandon transactions that have not been committed for longer than the specified timeout. |
restartTxWithDataTimeoutSec | Automatically restart DataJobs for transactions with data changes that have not been committed for longer than the specified timeout. |
oraUseOnlineDic | Whether to use online logs; false means using offline logs, which may put more stress on Oracle. |
oraReleaseIntervalSec | Interval for recreating connection for analysis to release Oracle server resources. |
oraMiningSessionPauseSec | Interval between execution of LogMiner commands for analysis. |
fallBackScnStep | Distance to keep from the latest Redo log data; 0 means to keep right behind the latest Redo log data. |
sqlCaseConversionEnabled | Whether to enable DDL case conversion (according to the default case rules of the current database). |
Tips: To modify the general parameters, see General Parameters and Functions.
Prerequisites
| Prerequisite | Description |
|---|---|
Permissions for Account | create, delete, create_index, delete_index, read, write permissions for indexes. |
Port Preparation | Allow the migration and sync node (Worker) to connect to the Elasticsearch port. |
Parameters
| Parameter | Description |
|---|---|
maxBulkSizeMb | Maximum batch size per table; If the batch size exceeds this limit, then data is flushed to the write queue. |
totalDataInMemMb | Maximum data size allowed in memory when writing in batches; If the data size exceeds the memory limit, or the wait time exceeds asyncFlushIntervalSec, then data is flushed to the write queue. |
asyncFlushIntervalSec | Interval to wait for flushing when writing in batches; If the wait time exceeds asyncFlushIntervalSec, or the data size exceeds totalDataInMemMb, then data is flushed to the write queue. |
realFlushPauseSec | Wait time to flush data to ElasticSearch using Bulk Write. 0 means no wait is needed. |
pkSeparator | Separator for concatenating _id (number of fields > 1). |
enableBulkSizeThreshold | Enable batch write mode (enabled by default). |
Tips: To modify the general parameters, see General Parameters and Functions.