AWS MSK Setup Guide Beta

Follow our setup guide to connect AWS MSK to Fivetran.

---

Prerequisites

To connect AWS MSK to Fivetran, you need an AWS MSK cluster.

---

Setup instructions

IMPORTANT: If you have enabled to the brokers of MSK clusters, skip to the .

(Optional) Configure AWS PrivateLink Beta

IMPORTANT: You must have a Business Critical plan to use AWS PrivateLink.

AWS PrivateLink allows VPCs and AWS-hosted or on-premises services to communicate with one another without exposing traffic to the public internet. PrivateLink is the most secure connection method. Learn more in .

Follow to configure PrivateLink for your AWS MSK platform.

For more information about configuring MSK clusters with AWS PrivateLink, see .

IMPORTANT: In PrivateLink connections, you don't need to expose brokers to public IP addresses. If you are using a PrivateLink connection, skip ahead to the .

Find ENI ID

1. Open the and go to your cluster.

2. In the Brokers summary section, for each broker find the Attached ENI ID. Make a note of the ENI ID(s).

   

Associate address

1. Open the and select Network Interfaces.

2. For every broker ENI ID you found in , select Actions > Associate address to associate an Elastic IP address.

   

3. Make a note of the associated Elastic IPs for each broker. You will need them to configure Fivetran.

Configure security groups

1. On the EC2 dashboard, select Security Groups.

2. In the Security Groups of the ENI IDs you found in , safelist .

Set property

Set the advertised.listeners property for the corresponding port of each broker to the Elastic IP associated with the ENI of the broker. Use to configure the broker-level properties.

(Optional - TLS) Download required certificates and passwords

1. Download the kafka.client.truststore.jks and kafka.client.keystore.jks files you created while . You will need them to configure Fivetran.

2. Make a note of the Your-Store-Pass and Your-Key-Pass passwords. You will need them to configure Fivetran.

   TIP: We recommend using the same value for Your-Store-Pass and Your-Key-Pass while configuring TLS.

Finish Fivetran configuration

1. In the , enter the Destination schema name of your choice.

2. Enter a dedicated Consumer Group name. Fivetran uses this consumer group, and the group name must be unique to each connector.

3. Enter your Servers details using the <host>:<port> format. Based on your connectivity type, do either of the following:

   • Public access: Enter the public broker endpoints from the AWS MSK Management Console. In the Cluster summary section, go to the View client information tab. Copy the Public endpoint information.

     TIP: Click + to add the host:port pairs. For example, if the endpoint is b-1-public.dummyclustername.abc123.c18.kafka.us-east-1.amazonaws.com:9092,b-4-public.dummyclustername.abc123.c18.kafka.us-east-1.amazonaws.com:9092, enter b-1-public.dummyclustername.abc123.c18.kafka.us-east-1.amazonaws.com:9092 and b-4-public.dummyclustername.abc123.c18.kafka.us-east-1.amazonaws.com:9092.

   • Non-public access: Use the Elastic IPs you found in as <host>. For example, if the Elastic IP for the broker is 10.100.10.10 and the authentication is SASL/SCRAM, then enter the server name as 10.100.10.10:9096.

     NOTE: Click + to add more than one set of server details.

4. Select your message type: Json, Avro, Protobuf, or Text.

   • If you choose Json, select a sync type: Packed or Unpacked.

   • If you choose Avro or Protobuf, enter the following schema registry credentials:

     • Schema Registry URLs
     • Schema Registry Key
     • Schema Registry Secret

5. Select a security protocol: PLAINTEXT, SASL/SCRAM, or TLS.

   • If you choose SASL/SCRAM, enter your secret's Username and Password.

   • If you choose TLS, provide the following details you found in :

     • In the Truststore File field, upload the kafka.client.truststore.jks file.
     • In the Keystore File field, upload the kafka.client.keystore.jks file.
     • Enter the Keystore Password.
     • Enter the Key Password.

6. Select your connection method: Connect directly or Connect via PrivateLink. If you choose Connect via PrivateLink, Fivetran connects to your message brokers using AWS PrivateLink.

7. Click Save & Test. Fivetran will take it from here and sync your AWS MSK data.

---

Related articles