High-Volume Agent Oracle
is an object-relational database management system created by the Oracle Corporation. Fivetran replicates data from your Oracle source database and loads it into your destination using .
NOTE: You must have an Enterprise or Business Critical plan to use the High-Volume Agent Oracle connector.
Supported services
The Fivetran High-Volume Agent connector supports the following Oracle database services:
Supported configurations
Fivetran supports the following Oracle configurations:
| Supportability OS | Database versions |
|---|---|
| Windows | 11.1, 11.2, 12.1, 12.2, 18c, 19c |
| Linux | 11.1, 11.2, 12.1, 12.2, 18c, 19c, Exadata |
| AIX | 11.2, 12.1, 12.2, 19c |
| Solaris | 11.1, 11.2, 12.1, 12.2, 19c |
Which Oracle instance types we can connect to depends on your database service.
| Instance Types | Supported |
|---|---|
| Primary Instance | check |
| Active Data Guard Standby Instance | check |
| Physical Standby Instance | |
| Real Application Cluster (RAC) | check |
| Multitenant (CDBs/PDBs) | check |
Known limitations
- Limited support for special* or lowercase characters in column and table names
- No support for special* or lowercase characters in schema names. For example,
lowercase_schema,lowercase_SCHEMA, orlOwErCaSeScHeMaare all unsupported names.
* Special characters include (, &, @, *, $, |, %, ~, -, and ). Underscores are not special characters.
Features
| Feature Name | Supported | Notes |
|---|---|---|
| check | All tables and fields | |
| check | All tables and fields | |
| check | Column level, table level, and schema level | |
| check | ||
| check | Table level | |
| check | ||
| check | ||
| Priority-first sync | ||
| Fivetran data models | ||
| Private networking |
Setup guide
Follow our step-by-step for specific instructions on how to set up your Oracle database with the High-Volume Agent connector.
Sync overview
Once Fivetran is connected to your Oracle source database, we pull a full dump of all selected data from your database. We then pull all your new and changed data at regular intervals using your online and archived redo logs. If data in the source changes (for example, you add new tables or change a data type), Fivetran automatically detects and persists these changes to your destination.
NOTE: Choosing a 1-minute sync frequency does not guarantee that your sync completes within one minute.
Replication speeds
Two major factors can cause disparities between our estimates and the exact replication speed for your Fivetran-connected databases: network latency and discrepancies in the format of the data we receive versus how the data is stored at rest in the data destination.
The ability to sync changes quickly also depends on the sync frequency you configure. The risk of the sync falling behind, or being unable to keep up with data changes, decreases as the sync frequency increases. We recommend a higher sync frequency for data sources with a high rate of data changes.
To measure the rate of new data in your database, check the disk space usage metrics over time for databases hosted on cloud providers. For self-hosted databases, you can run the following query to determine disk space usage:
SELECT SUM(bytes)/1024/1024 AS MB FROM dba_segments;
content_copySchema information
Fivetran tries to replicate the exact schema and tables from your Oracle source database to your destination according to our . We create schemas in your destination that map directly to the schemas in your source Oracle database, ensuring that the data in your destination is in a familiar format to work with.
When you connect to Fivetran and specify a source database, you also select a schema prefix. We map the schemas we discover in your source database to your destination and prepend the destination schema name with the prefix you selected.
Fivetran-generated columns
Fivetran adds the following columns to every table in your destination:
_fivetran_deleted(BOOLEAN) marks rows that were deleted in the source database._fivetran_synced(UTC TIMESTAMP) indicates the time when Fivetran last successfully synced the row._fivetran_id(STRING) is a unique ID that Fivetran uses to avoid duplicate rows in tables that do not have a primary key.
We add these columns to give you insight into the state of your data and the progress of your data syncs.
Type transformations and mapping
As we extract your data, we match Oracle data types to types that Fivetran supports. If we don't support a certain data type, we automatically change that type to the closest supported type or, in some cases, don't load that data at all. Our system automatically skips columns of data types that we don't accept or transform.
The following table illustrates how we transform your Oracle data types into Fivetran supported types:
| Oracle Type | Fivetran Type | Fivetran Supported | Notes |
|---|---|---|---|
| CHAR | STRING | True | |
| NCHAR | STRING | True | |
| VARCHAR | STRING | True | |
| VARCHAR2 | STRING | True | |
| NVARCHAR2 | STRING | True | |
| NUMBER | One of the following types based on the source value's precision: - Short integers (precision < 5) to SHORT - Integers (precision < 10) to INTEGER - Long integers (precision < 19) to LONG - Integers (precision < default value) to BIGDECIMAL | True | We map floating-point, double-precision floating-point, or big decimal numbers to the BIGDECIMAL data type. If we detect precision or scale larger than your default values, we map that data to the STRING data type. |
| FLOAT | BIGDECIMAL | True | |
| DOUBLE PRECISION | BIGDECIMAL | True | |
| REAL | BIGDECIMAL | True | |
| DATE | LOCALDATE | True | We map DATE columns to LOCALDATETIME if the values have non-zero time information. |
| TIMESTAMP | LOCALDATETIME | True | |
| TIMESTAMP WITH TIME ZONE | INSTANT | True | |
| TIMESTAMP WITH LOCAL TIME ZONE | INSTANT | True | |
| RAW | BINARY | True | |
| BINARY_FLOAT | FLOAT | True | |
| BINARY_DOUBLE | DOUBLE | True | |
| INTERVAL YEAR TO MONTH | False | ||
| INTERVAL DAY TO SECOND | False | ||
| LONG | STRING | True | We truncate LONG columns if they exceed your destination's maximum allowed length.* |
| LONG RAW | False | ||
| ROWID | False | ||
| UROWID | False | ||
| BFILE | False | ||
| BLOB | BINARY | True | We truncate LOB columns if they exceed your destination's maximum allowed length.* |
| CLOB | STRING | True | We truncate LOB columns if they exceed your destination's maximum allowed length.* |
| NCLOB | STRING | True | We truncate LOB columns if they exceed your destination's maximum allowed length.* |
* Learn more in our documentation.
If we are missing an important type that you need, please .
In some cases, when loading data into your destination, we may need to convert Fivetran data types into data types that are supported by the destination. For more information, see the .
Excluding source data
If you do not want to sync all the data from your source database, you can exclude schemas, tables, or columns from your syncs on your Fivetran dashboard. To do so, go to your connector details page and uncheck the objects you would like to omit from syncing. For more information, see our .
You can also use SELECT permissions if you want to exclude specific schemas and tables from syncing. The Fivetran user can only discover schemas and tables for which they have been given explicit SELECT permission. Note that permissions do not shield confidential data, such as PII (Personally Identifiable Information), from Fivetran because they do not apply to archived redo logs. Via the archived redo log files, Fivetran has access to the full contents of any changed rows of all tables that pass through your system. However, we filter out the prohibited data at the earliest possible stage of our syncs and do not load it into your destination.
Initial sync
When Fivetran connects to a new database, we first copy all rows from every table in every schema for which we have SELECT permission (except for those you excluded in your Fivetran dashboard) and add . We copy rows by performing a SELECT statement on each table. We import a maximum of four tables at a time, before moving onto the next, until all is complete.
To keep your data up to date after the initial sync, we use log-based capture. This allows Fivetran to capture and then update only the data that has changed since our last sync.
Updating data
Fivetran performs incremental updates of any new or modified data from your source database. We use Direct Capture, a proprietary replication method, to extract your database's change data directly from the log files.
Tables with a primary key
We merge changes to tables with primary keys into the corresponding tables in your destination:
- An INSERT in the source table generates a new row in the destination with
_fivetran_deleted = FALSE. - A DELETE in the source table updates the corresponding row in the destination with
_fivetran_deleted = TRUE. - An UPDATE in the source table updates the data in the corresponding row in the destination.
If we detect that your primary key has changed, we handle DELETEs and UPDATEs differently:
- If you have set
SUPPLEMENTAL LOG DATA (ALL)on your source table, we update the row with the old primary key value with_fivetran_deleted = TRUE. We then insert a row with the new primary key value with_fivetran_deleted = FALSE. - If you have not set
SUPPLEMENTAL LOG DATA (ALL)on your source table, we do not update the row with the old primary key value because we can't identify it. We insert a row with the new primary key value with_fivetran_deleted = FALSE.
NOTE: If we detect that your primary key has changed, and you have not set
SUPPLEMENTAL LOG DATA (ALL)on your source table, we raise a warning telling you to setSUPPLEMENTAL LOG DATA (ALL)on the table.
Tables without a primary key
For tables without a primary key, we designate our _fivetran_id column as the primary key. We populate this column with Oracle's ROWID pseudo column value.
NOTE: For a primary keyless table with row movement enabled, any update operation may incur additional MAR if it causes a row movement. When the row movement occurs, Oracle generates the following three log events, and Fivetran counts them as two MAR:
UPDATE (to an existing row) DELETE (the row from the current partition) INSERT (the row into a new partition)content_copy
- If the table is partitioned and row movement is enabled, we use a hash of the column's non-Fivetran values to populate our
_fivetran_idcolumn. - Otherwise, we use Oracle's
ROWIDpseudo column to populate our_fivetran_idcolumn.- If you enable row movement later, we the table and use a hash of the column's non-Fivetran values to populate our
_fivetran_idcolumn. - If we detect any DDL operations (excluding
GRANT), we the table and re-import its contents with the newly assignedROWIDs.
- If you enable row movement later, we the table and use a hash of the column's non-Fivetran values to populate our
Deleted rows
We do not delete rows from your destination. When a row is deleted from the source table, we set the _fivetran_deleted column value of the corresponding row in the destination to TRUE.
Deleted columns
We do not delete columns from your destination. When a column is deleted from the source table, we replace the existing values in the corresponding destination column with NULL values.
Excluded tables
Fivetran does not sync the following tables:
- Temporary tables
- System-generated supplemental tables (for example, MLOG$ tables)
- System tables (for example, those in the
SYSTEMtablespace or those owned bySYSuser)
Comments
0 comments
Please sign in to leave a comment.