Firebase Beta

is an application development platform. Firebase Cloud Firestore is a NoSQL document database characterized by a lack of a fixed schema. Data is stored in key-value pairs in documents that form a collection.

Supported services

Fivetran supports the Firebase .

NOTE: We support only the Cloud Firestore databases in Native mode.

Supported configurations

Fivetran supports the following Firebase configurations:

Supportability Category	Supported Values
Connector limit per database	No limit

Features

Feature Name	Supported	Notes
		All tables and fields
		All tables and fields
		Column level, table level, and schema level

		Table level
History

Priority-first sync
Fivetran data models
Private networking

Setup guide

For specific instructions on how to set up your Firebase connector, see the .

Sync overview

Once Fivetran is connected to your Firestore database, we pull a complete dump of all selected data from your database. The initial sync finishes when all collections that existed when the sync started have finished importing. In each sync, we pull all the data from the source and find the difference between the syncs to get the updated data.

Packing mode options

Packing modes determine the form in which Fivetran delivers your data. You must choose a packing mode for each table you want Fivetran to sync. There are two packing modes - packed and unpacked. You must use packed mode if your table has or will have more than 1000 unique first-level fields.

Unpacked mode

If your table has fewer than 1000 first-level fields, you can have your data delivered unpacked. Fivetran unpacks one layer of nested fields and infers types.

In unpacked mode, the following source table

{
 "foo": 1, <== document_id
 "bar": 2,
 "nested": {
   "baz": 3
 }
}
content_copy

is delivered to your destination as

foo INTEGER	bar INTEGER	nested JSON
1	2	`{"baz":3}`

Packed mode

All tables with more than 1000 first-level fields must have their data delivered packed, though you can select the packed mode for smaller tables too.

In packed mode, the following source table

{
 "foo": 1, <== document_id
 "bar": 2,
 "nested": {
   "baz": 3
 }
}
content_copy

is delivered to your destination as

foo INTEGER	data JSON
1	`{"foo":1, "bar":2, nested":{"baz":3}}`

Switching pack modes

You can switch packing modes for a table at any time in your Fivetran dashboard. When you change the packing mode for a table, we automatically perform a full table re-sync.

To change the packing mode for a table, do the following:

Go to the Setup tab in the connector dashboard.
Click Edit connection details.
In the connector setup form, change the Packing Mode. If you selected packed mode, select the table(s) you want to sync. Any tables that you do not select will be synced in default mode.
Click Save & Test.

Replication speeds

We pull all the data from source in each sync, so there might be some delay to fetch and process this data.

Two major factors can cause disparities between our estimates and the exact replication speed for your Fivetran-connected databases: network latency and discrepancies in the format of the data we receive versus how the data is stored at rest in the destination. The ability to sync changes quickly also depends on your configured sync frequency. We recommend setting up a higher sync frequency or frequency close to your average sync speed for data sources with a high rate of data changes.

Schema information

Fivetran tries replicating the exact schema and tables from your Firestore database to your destination.

Fivetran-generated columns

Fivetran adds the following columns to every table in your destination:

_fivetran_deleted (BOOLEAN) marks deleted rows in the source database.
_fivetran_synced (UTC TIMESTAMP) indicates when Fivetran successfully synced the row.

We add these columns to give you insight into the state of your data and the progress of your data syncs.

Type transformations and mapping

As we extract your data, we match Firestore data types to types that Fivetran supports. If we don't support a data type, we automatically change that type to the closest supported type or, in some cases, don't load that data at all. Our system fails when we encounter columns with data types that we don't accept or transform.

The following table illustrates how we transform your Firestore data types into Fivetran supported types:

Firestore Data Type	Fivetran Data Type	Fivetran Supported
Array	JSON	True
Boolean	BOOLEAN	True
Date and time	INSTANT	True
Floating-point number	DOUBLE	True
Geographical point	STRING	True
Integer	LONG	True
Map	JSON	True
Null	NULL	True
Reference	STRING	True
Text string	STRING	True

In some cases, when loading data into your destination, we may need to convert Fivetran data types into data types that are supported by the destination. For more information, see the .

Excluding source data

If you don’t want to sync all the data from your master database, you can exclude schemas or tables from your syncs on your Fivetran dashboard. To do so, go to your connector details page and uncheck the objects you would like to omit from syncing. For more information, see our .

Alternatively, you can change the permissions to restrict access to particular collections or sub-collections using .

Initial sync

When Fivetran connects to a new Firestore database, we first copy all data from every collection (except for those you have excluded in your Fivetran dashboard) and add . We copy data by performing a db.collection(collection).get() operation on each collection. We do not pull entire data, we paginate through the result to make sync failure tolerant.

Updating data

Fivetran pulls all the data from your source in each sync. We calculate the difference between current and previous sync to identify updates and deletes, and then we process them accordingly.

Fivetran uses Firestore's built-in document_id (custom or auto-generated) and write it as _id as primary key of the table. Using the _id field to identify rows, we merge changes to your documents into the corresponding tables in your destination:

Every inserted row in the source generates a new row in the destination with _fivetran_deleted = FALSE.
Every updated row in the source updates the data in the corresponding row in the destination, with _fivetran_deleted = FALSE.
For every deleted row, the _fivetran_deleted column value is set to TRUE for the corresponding row in the destination.

Fivetran Teleport Sync

Fivetran Teleport Sync is a proprietary database replication method that offers the completeness of snapshots, which is a substitute of Snapshot Listener. With this sync mechanism, Fivetran can incrementally replicate your database with no additional setup other than connecting to your database.

Fivetran Teleport Sync's queries perform the following operations on your database:

Pull all the data of each selected table
Perform calculations on all values in each synced table's rows
Aggregate a compressed table snapshot in the database's memory

Fivetran Teleport Sync performs following actions in each sync:

New updates on each row
Capture deletes
Data type changes that do not affect existing values (For example, Number to String)

Deleted rows

We do not delete rows from your destination. When a row is deleted from the source table, we set the _fivetran_deleted column value of the corresponding row in the destination to TRUE.

Excluded tables

We sync nested collections or sub-collections present up to Level 1 (level 1). We do not sync any nested data Level 2 (level 2) onwards.

collection:(level 0)
    document:
        Id:1
        name:foo
        nested_collection:(level 1)
            nested_document:
                Id:2
                name:nested_foo
                nested_collection_2:(level 2)
                    nested_document_2:
                        Id:nested_2
                        name:nested_level_2_foo
content_copy