Databricks as Target

Fivetran Local Data Processing supports integrating changes into Databricks location. This section describes the configuration requirements for integrating changes using and into Databricks location. For the list of supported Databricks versions into which Local Data Processing can integrate changes, see in .

The preferred methods for writing data into Databricks are using and as they provide better performance. However, it is required to create staging files on a temporary location to perform and . For more information about staging, see section .

Dropping Columns from Delta Tables

To drop column(s) from delta tables the column mapping feature must be enabled manually in Databricks. For more information, refer to in .

If column mapping is enabled and when a column is dropped in the source location, Local Data Processing will perform ALTER TABLE operation on the delta table.

If column mapping is not enabled then Local Data Processing will recreate the delta table using (with create table option).

Time Zone

The time zone of the High-Volume Agent and the Databricks database must be the same. If they are not the same, the following environment variables and HVR_SQL_INIT should be defined to make them same:

Group	Table	Action	Parameter(s)
Databricks	*		=TZ =time_zone
Databricks	*		=HVR_SQL_INIT =SET TIME ZONE 'time_zone'

The environment variable HVR_SQL_INIT runs the SQL statement when the database connection is established.

Limitations

For Databricks runtime version earlier than 11.2, the DEFAULT constraint is not supported. As a result, it cannot automatically provide a default value for a column when changing its nullability. Therefore, if you need to alter a nullable column to NOT NULL in the target, you must first ensure that all existing NULL values in the source are updated with non-NULL values.