Architecture

Fivetran connects to all of your supported data sources and loads the data from them into your destination. Each data source has one or more connectors that run as independent processes that persist for the duration of one update. A single Fivetran account, made up of multiple connectors, loads data from multiple data sources into one or more destinations.

Fivetran System Architecture Overview Diagram:

---

Connect

Fivetran connects to your data sources using our connectors. Fundamentally, there are two different types of connectors: push and pull.

Pull connectors

Fivetran's pull connectors actively retrieve, or pull, data from a source. Fivetran connects to and downloads data from a source system at a fixed frequency. We use an SSL-encrypted connection to the source system to retrieve data using a variety of methods: database connections via ODBC/JDBC, or web service APIs via REST and SOAP. In practice, the method or combination of methods is different for every source system.

Push connectors

In push connectors, such as or , source systems send data to Fivetran as events. The push connector pipeline is as follows:

1. When we receive the events in our collection service, we first buffer the events in the queue.
2. We store the event data as JSON in our cloud storage buckets. (For more information, see our .)
3. During the sync, we push the data to your destination.

For more information on how sync works in our push connectors, see .

Ingest and Prepare Data

Once the connector process ingests the data query results, Fivetran normalizes, cleans, sorts, and de-duplicates the data. The purpose of the normalization and cleaning is to format the data in the optimal way for the destination. To learn more about this optimization, see our .

The Fivetran philosophy is to make a faithful replication of source data with as few transforms as necessary to make it useful.

Fivetran uses a queue to buffer the incoming source data. When failures are caused by transient errors or destination unavailability, Fivetran’s pipeline doesn't attempt to retrieve the data from the source that we already have in our queue. This limits the impact of destination outages and improves Fivetran reliability. When we find unprocessed data in the storage queue because of destination load failures, we process the pending queued data first.

During the ingestion process, we that is encrypted at rest using a secret ephemeral key until we successfully load the data into the destination.

The ingestion processes run in parallel with the preparation and load processes. This strategy ensures that the destination data load process doesn't block the source data ingestion process.

Load Data into Temporary Data Storage

Fivetran outputs the finalized records to a file in a file storage bucket. We encrypt this file with a separate ephemeral key that is known only to the data writing process. We automatically delete this temporary file after 7 days using an expiration policy on the bucket.

The bucket depends on the following factors:

• For , your selected cluster region
• For other types, and if you don't choose to create Delta tables in an external location for Databricks:
  • your plan
  • selected data processing region
  • selected cloud service provider (CSP)
  • selected CSP region

See our for details.

Write Data into Destination

From the temporary data storage, Fivetran copies the file into staging tables in the destination. In the process, we transmit the ephemeral encryption key for the file to the destination so it can decrypt the data as it arrives. Before we write the data into the destination, we update the schema of existing tables to accommodate the newer incoming batch of data. We then merge the data from the staging tables with the existing data present in the destination. Finally, we apply the deletes (if any) on the existing tables. Once we complete the write process, the connector process terminates itself. A system scheduler will later restart the process for the next update.