Databricks Setup Guide

Follow our setup guide to connect Databricks to Fivetran.

IMPORTANT: If you used Databricks Partner Connect to set up your Fivetran account, you don't need to follow the setup guide instructions because you already have a connection to Databricks. We strongly recommend using Databricks Partner Connect to set up your destination. Learn how in . To connect your Databricks workspace to Fivetran using Partner Connect, make sure you meet .

Prerequisites

To connect Databricks to Fivetran, you need the following:

a Databricks account
a Fivetran account with
enabled on your Databricks Workspace. Unity Catalog is a unified governance solution for all data and AI assets including files, tables, machine learning models and dashboards in your lakehouse on any cloud. We strongly recommend using Fivetran with Unity Catalog as it simplifies access control and sharing of tables created by Fivetran. Legacy deployments can continue to use Databricks without Unity Catalog.
. SQL warehouses are optimized for data ingestion and analytics workloads, start and shut down rapidly and are automatically upgraded with the latest enhancements by Databricks. Legacy deployments can continue to use with Databricks Runtime v7.0+.

Databricks on AWS - Setup instructions

Choose a catalog

If you don't use Unity Catalog, skip ahead to the . Fivetran will create schemas in the default catalog, hive_metastore.

If you use Unity Catalog, you need to decide which to use with Fivetran. For example, you could create a catalog called fivetran and organize tables from different connectors in it in separate schemas, like fivetran.salesforce or fivetran.mixpanel. If you need to set up Unity Catalog, follow Databricks' .

Log in to your Databricks workspace.
Click Data in the Databricks console.
Choose a catalog in the Data Explorer.

Connect SQL warehouse

In the Databricks console, go to SQL > SQL warehouses > Create SQL warehouse. If you want to select an existing SQL warehouse, skip to step 5 in this section.
In the New SQL warehouse window, enter a Name for your warehouse.
Choose your Cluster Size and configure the other warehouse options.
Click Create.
Go to the Connection details tab.
Make a note of the following values. You will need them to .
- Server Hostname
- Port
- HTTP Path

Create personal access token

Fivetran uses a secure token to connect to Databricks. Follow .

(Optional) Connect using AWS PrivateLink Beta

IMPORTANT: You must have a Business Critical plan to use AWS PrivateLink.

You can connect Fivetran to your Databricks destination using AWS PrivateLink. AWS PrivateLink allows VPCs and AWS-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. PrivateLink is the most secure connection method. Learn more in .

How it works:

Fivetran accesses the data plane in your AWS account using the control plane network in Databricks' account.
You set up a back-end AWS PrivateLink connection between your AWS account and Databricks' AWS account (shown as (Workspace 1/2) Link-1 in the diagram above).
Fivetran creates and maintains a front-end AWS PrivateLink connection between Fivetran's AWS account and Databricks' AWS account (shown as Regional - Link-2 in the diagram above).

Prerequisites

To set up AWS PrivateLink, you need:

A Fivetran instance configured to run in AWS
A Databricks destination in one of
All of

Configure AWS PrivateLink

Follow Databricks' to enable private connectivity for your workspaces. Your workspaces must have the following:

A registered back-end VPC endpoint for secure cluster connectivity relay
A registered back-end VPC endpoint for REST APIs
A PAS object with access to Fivetran's VPC endpoints
If the Private Access Level on the PAS object is set to Account, a Fivetran VPC endpoint (for the applicable AWS region) that's registered once per account
If the Private Access Level on the PAS object is set to Endpoint, a Fivetran VPC endpoint (for applicable AWS region) that's registered using the allowed_vpc_endpoint_ids property

Register Fivetran endpoint details

Register the Fivetran endpoint for the applicable AWS region with your Databricks workspaces. We cannot access your workspaces until you do so.

AWS Region	VPC endpoint
ap-south-1 Asia Pacific (Mumbai)	`vpce-089f13c9231c2b729`
ap-southeast-2 Asia Pacific (Sydney)	`vpce-0e5f79a1613d0cf05`
ca-central-1 Canada (Central)	`vpce-09f0049f9a92177f1`
eu-central-1 Europe (Frankfurt)	`vpce-049699737170c880d`
eu-west-1 Europe (Ireland)	`vpce-0b32cb6c08f6fe0df`
eu-west-2 Europe (London)	`vpce-03fde3e4804f537eb`
us-east-1 US East (N. Virginia)	`vpce-0ff9bd04153060180`
us-east-2 US East (Ohio)	`vpce-05153aa99bf7a4575`
us-west-2 US West (Oregon)	`vpce-0884ff0f23dcbf0dc`

IMPORTANT: Regardless of your Fivetran subscription plan, if you have enabled back-end AWS PrivateLink connection between your AWS account and Databricks' AWS account (shown as (Workspace 1/2) Link-1 in the diagram above), you must register the Fivetran endpoint (for the applicable AWS region) to avoid connection failures.

Complete Fivetran configuration

Log in to your Fivetran account.
Go to the , then click + Add Destination.
On the Add destination to your account page, enter a Destination name of your choice.
Click Add.
Select Databricks as the destination type.
(Unity Catalog only) Enter the Catalog name. This is the name of the catalog in Unity Catalog, such as fivetran or main.
Enter the Server Hostname.
NOTE: If we auto-detect your Databricks Deployment Cloud, the Databricks Deployment Cloud field won't be visible in the setup form.
Enter the Port number.
Enter the HTTP Path.
Enter the Personal Access Token you created in the .
(Optional) Choose the Databricks Deployment Cloud based on which cloud you are using with Databricks.
(Optional) Set the Create Delta tables in an external location toggle to ON to create Delta tables as external tables. You can:
- Enter the External Location you want to use. We will create the Delta tables in the {externallocation}/{schema}/{table} path.
- Use the default location registered with the cluster. Do not specify the external location. We will create the external Delta tables in the /{schema}/{table} path.
Choose your Connection Method:
- Connect directly
- Connect via SSH
- Connect via PrivateLink
NOTE: The Connect via PrivateLink option is only available for Business Critical accounts.
Choose the Data processing location. Depending on the plan you are on and your selected cloud service provider, you may also need to choose a Cloud service provider and AWS region as described in our .
Choose your Time zone.
(Optional for Business Critical accounts) To enable , set the Use Failover toggle to ON, and then select your Failover Location and Failover Region. Make note of the IP addresses of the secondary region and safelist these addresses in your firewall.
Click Save & Test.

(Optional) Storing data in external locations

With Unity Catalog

You can customize where Fivetran stores Delta tables. If you use Unity Catalog, Fivetran creates managed tables in Databricks. Managed tables are stored in the root storage location that you configure when you create a metastore.

You can also instruct Fivetran to store tables in an managed by Unity Catalog. To do so, follow these steps:

Follow Databricks' to add a storage credential to Unity Catalog.
Follow Databricks' to add an external location to Unity Catalog.
Enable Create Delta tables in an external location and specify the external location (for example, s3://mybucket/myprefix ) as the value.

Without Unity Catalog

If you do not use Unity Catalog, you can still store Delta tables in a specific S3 bucket. First, you must create an AWS instance profile and associate it with Databricks compute.

See Databricks' . Perform the first four steps mentioned in the guide to create an instance profile.
In the Databricks console, click Settings > SQL Admin Console.
In the Settings window, select SQL Warehouse Settings.
Select the Instance profile you created above.

(Optional) Connect Databricks cluster

TIP: We strongly recommend using Databricks SQL warehouses with Fivetran. To learn more, skip to the .

To connect to a Databricks cluster, do the following:

Create a Databricks cluster

Log in to your Databricks workspace.
In the Databricks console, go to Data Engineering > Cluster > Create Cluster..
Enter a Cluster name of your choice.
Set the Databricks Runtime Version to the latest LTS release. At minimum, you must choose v7.3+.
Select the Cluster mode.
Set the Databricks Runtime Version to 7.3 or later. (10.4 LTS Recommended)
(Optional) If you are using the Unity Catalog feature, in the Advanced Options window, in the Security mode drop-down menu, select either Single user or User isolation.
(Optional) If you are using an external data storage and have disabled the Unity Catalog feature, select the Instance profile you created in the .
In the Advanced Options section, select Spark.

If you have set the Databricks Runtime Version to below 9.1, paste the following code in Spark config field:

spark.hadoop.fs.s3a.impl shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3n.impl shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3n.impl.disable.cache true
spark.hadoop.fs.s3.impl.disable.cache true
spark.hadoop.fs.s3a.impl.disable.cache true
spark.hadoop.fs.s3.impl shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem
content_copy

set-aws-true

Click Create Cluster.
In the Advanced Options window, select JDBC/ODBC.

jdbc-properties

Make a note of the following values. You will need them to .
- Server Hostname
- Port
- HTTP Path

For further instructions, skip to the .

Databricks on Azure - Setup instructions

(Optional) Configure Unity Catalog

IMPORTANT: Perform the following steps if you want to use the Unity Catalog feature or an external data storage.

Create a storage account

To create an Azure Blob Storage account, follow Microsoft's .
To create an Azure Data Lake Storage Gen2 storage account, follow Microsoft's .

Create a container

To create a container in Azure Blob storage, follow Microsoft's .
To create a container in ADLS Gen2 storage, follow Microsoft's .

Configure a managed identity for Unity Catalog

IMPORTANT: Perform this step if you have enabled the Unity Catalog feature and want to access the metastore using a managed identity.

You can configure Unity Catalog (Preview) to use an Azure managed identity to access storage containers.

To configure a managed identity for Unity Catalog, follow Databricks' .

Create a metastore and attach workspace

IMPORTANT: Perform this step if you have enabled the Unity Catalog feature.

To create a metastore and attach workspace, follow Microsoft's .

(Optional) Connect using Azure Private Link Beta

IMPORTANT: You must have a Business Critical plan to use Azure Private Link.

You can connect Fivetran to your Databricks destination using Azure Private Link. Azure Private Link allows VNet and Azure-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. Learn more in .

Prerequisites

To set up Azure Private Link, you need:

A Fivetran instance configured to run in Azure
A Databricks destination in one of
An Azure Databricks workspace that is in your own virtual network (Vnet injection). Learn more in .

IMPORTANT: Fivetran cannot connect Private Link to an Azure Databricks workspace that's spun using default deployments.

Configure Azure Private Link

Create or select an Azure Databricks workspace that was spun using the Vnet injection deployment method.
Send your ID and Workspace URL to your Fivetran account manager. Fivetran sets up the Azure Private Link connection on our side.
Once your account manager confirms our setup was successful, approve our endpoint connection request. Setup is now complete.