Confluent Cloud

Nexla's bi-directional connectors can both send data to and receive data from any data system. This means that once a user has created or gained access to a credential for any data system, building any data flow to ingest data from or send data to a location within that data system requires only a few simple steps.

1. Confluent Cloud Credentials

This section provides information about and step-by-step instructions for creating a new Confluent Cloud credential in Nexla.

1.1. Add a New Confluent Credential

1. After selecting the data source/destination type, in the screen, click . This will open the Add New Credential window.

2. Enter a name for the credential in the Credential Name field.

3. Optional: Enter a description for the credential in the Credential Description field.

1.2 Configure the Confluent Cloud Environment Settings

1. Enter the address of the Confluent bootstrap servers to which Nexla should connect in the Boostrap Servers field. This is usually in the form of a Virtual IP address.

2. Enter the Confluent Cloud API Key required to connect to the Confluent Cloud cluster in the Confluent Cloud API Key field. This API key is found under the account settings in Confluent.

3. Enter the Confluent Cloud API secret required to connect to the Confluent Cloud cluster in the Confluent Cloud API Secret field. This API secret is found under the account settings in Confluent.

4. Optional: Enter the consumer group ID to which Nexla users who use this credential will belong in the Consumer Group ID field.

5. Section 1.3 provides instructions for configuring Nexla to read schemas from the schema registry when using this credential, and Section 1.4 provides information about advanced settings available for Confluent Cloud credentials along with step-by-step instructions for configuring each setting.

   • To configure Nexla to read schemas from the schema registry, continue to Section 1.3.
   • To configure any desired additional advanced settings for this credential, continue to Section 1.4, and complete the relevant steps.
   • To create this credential without configuring any advanced settings, continue to Section 1.5.
     
     

1.3 Optional: Configure the Schema Registry

1. To read schemas from the schema registry when using this Confluent Cloud credential, check the box next to "Should Read Schemas from Schema Registry?".

2. Enter the schema registry URL for the Confluent schema registry in the Schema Registry URL field.

3. Enter the API key generated for this schema registry in the Schema Registry API Key field.

4. Enter the API secret generated for this schema registry in the Schema Registry API Secret field.

5. • To configure any desired additional advanced settings for this credential, continue to Section 1.4, and complete the relevant steps.
   • To create this credential without configuring any advanced settings, continue to Section 1.5.
     
     

1.4 Advanced Settings

1. Click at the bottom of the Add New Credential window to access additional available settings for the Confluent credential.

2. If the data from Confluent Cloud from which data should be read is not publicly accessible, check the box next to . This will append additional related fields to be populated in the Add New Credential window.

   SSH Tunnel

   Selecting this option allows Nexla to connect to a bastion host via SSH, and the database connection will then be provided through the SSH host.

3. Enter the SSH tunnel hostname or IP address of the bastion host running the SSH tunnel server that has access to the database in the SSH Tunnel Host field.

4. Enter the port of the tunnel bastion host to which Nexla will connect in the SSH Tunnel Port field.

5. Create an SSH username for Nexla in the bastion host, and enter that username in the Username for Tunnel field.

   SSH Username

   Usually, the username is set as "nexla".

1.5 Save and Create the Confluent Cloud Credential

1. Once all of the relevant steps in the above sections have been completed, click at the bottom of the Add New Credential screen to save the credential and all entered information.

2. The newly added credential will now appear in a tile on the screen and can be selected for use with a new data source or destination.

2. Data Source: Ingesting Data from Confluent Cloud

With Nexla's connectors, users can quickly and easily add Confluent Cloud as a data source to begin ingesting, transforming, and moving data in any format.

1. After logging into Nexla, navigate to the Integrate section by selecting from the platform menu on the left side of the screen.

2. Click at the top of the Integrate toolbar on the left to open the Select Source Type screen.

3. Search for or scroll to the Confluent connector, and click on the connector to select it.

4. Create a new credential that Nexla should use to connect to Confluent Cloud, or select an existing credential, and click . This will open the data source configuration screen.

   Creating a Confluent Credential

   Detailed instructions for creating a new Confluent credential and configuring credential settings in Nexla are provided in Section 1 above.

5. In the screen, under Configure Confluent Source, use the Select Confluent Topic to Read From pulldown menu to select the topic containing data that should be ingested from Confluent Cloud.

6. Select the default offset mode that will be used for this data source from the Default Offset Mode pulldown menu. The available options are Earliest, From Date, Latest, and Manual.

7. Select the data format parser for the data in the topic from the Data Format pulldown menu.

   Data Format

   The default setting for the data format parser is JSON.

8. Click to fetch a preview of the data that Nexla will pull from the topic.

9. Once all of the above steps have been completed, click in the upper right corner of the screen to save and create the new Confluent Cloud data source in Nexla.

   • Nexla will now begin scanning the topic for data and will organize any data that it finds into one or more Nexsets.
     
     

2.1 How Nexla Organizes Data

When Nexla ingests data from a source—whether a file-based storage system or any other type of service—the platform intelligently analyzes the structure of the data to organize it into one or more Nexsets.

If a location containing multiple files is selected when configuring a data source from a file-based storage system, Nexla will examine the differences between the ingested files. The platform will create Nexsets containing the ingested data based on the level of overlap between records and options selected during data source creation.

After the initial data ingestion cycle, Nexla will repeat the process of comparing the structure and composition of data newly ingested in subsequent cycles to any existing Nexsets. Similar data will be added to existing Nexsets, while significantly different data will be organized into a new Nexset.

Important Note: Data Ingestion & Processing

Nexla's comparison of ingested data to existing Nexsets ignores differences in file format. For example, when a CSV file containing the headers "ID" and "Name" and a JSON file with "ID" and "Name" object properties are ingested, the data contained in both files will be processed into the same Nexset.

2.2 Ingestion of New and/or Modified Files

Once a data source has been created in Nexla from Confluent Cloud, the platform will scan the source at regular intervals according to the configured scheduling options. When Nexla detects new files during a scan, it will automatically ingest and process the data contained in the new files and mark the files as ingested.

Nexla also tracks the number of rows of data that have been ingested from each file. Therefore, when additional rows of data are added to a previously ingested file, the platform will automatically ingest and process the added data.

Important Note: Data Ingestion Scheduling

Nexla reads and processes data from a source according to the configured schedule, but the platform will wait for a period of inactivity at the source.

3. Data Destination: Sending Data to Confluent Cloud

Nexla's bi-directional connectors allow data to flow both to and from any location, making it simple to set up a data flow that sends data to a database or data warehouse. This section provides general instructions and information about sending data to databases and data warehouses.

3.1 Select Confluent as a Destination

1. Locate the Nexset that will be sent to the Confluent Cloud destination.

   Locating Nexsets

   • To view all accessible Nexsets within their associated data flows, navigate to the Integrate section by selecting from the platform menu on the left side of the screen. In the All Data Flows screen, click on a listed data flow to view all detected and transformed Nexsets that it contains.

   • To view a list of all accessible Nexsets, select from the platform menu, and select from the Integrate menu on the left to open the Nexsets screen.

   
   

2. Click the icon on the Nexset. This will open the Send Nexset to Destination screen.

3. Select the Confluent connector—. Then, click in the top right corner of the screen.

   • This will open the page, where users can input and/or select credential information.

4. Select the credential that Nexla should use to connect to the database, and click in the top right corner of the screen. This will open the data destination configuration screen.

   Creating a Confluent Credential

   Detailed instructions for creating a new Confluent credential and configuring credential settings in Nexla are provided in Section 1 above.

5. With Nexla, data contained in Nexsets can be sent to an existing topic in Confluent Cloud, or users can create and send the data to a new topic.
   • To learn how to send data to an existing topic, see Section 3.2.
   • To learn how to create and send data to a new topic, see Section 3.3.
     
     

3.2 Send Data to an Existing Topic

1. In the screen, under Configure Destination, find the topic to which Nexla should send the data. Expand topics as necessary by clicking the icon next to each.

   Preview Data in the Topic

   To preview the data contained in a listed topic, hover over the topic listing, and click the button that appears to the right.

2. Use the Or Select Exsiting Topics pulldown menu to find the topic to which the data should be sent, and select the topic by hovering over it and clicking the button that appears to the right.

   • The button should now display , and the path of the selected topic will be shown at the top of the list.
   

3. Optional: To include Nexla metadata in the message sent to the Confluent topic, check the box next to "Include Nexla Metadata in Message?".

4. Optional: Select an attribute to be used as the Kafka message key from the list of attributes in the Select Attribute to be Used as Kafka Message Key pulldown menu.

5. Continue to Section 3.4.
   
   

3.3 Create and Send Data to a New Topic

1. In the screen, under Configure Destination, check the box next to Create a new topic.

2. Enter a name for the new topic in the Topic Name field.

3. Optional: To include Nexla metadata in the message sent to the Confluent topic, check the box next to "Include Nexla Metadata in Message?".

4. Optional: Select an attribute to be used as the Kafka message key from the list of attributes in the Select Attribute to be Used as Kafka Message Key pulldown menu.

5. Continue to Section 3.4.
   
   

3.4 Save and Activate the Confluent Cloud Destination

1. Once all of the above steps have been completed, click in the top right corner of the screen.

2. The confirmation screen indicates that the configured database destination has been successfully created in the data flow.

3. Optional: To edit the name of the newly created destination in this screen, click on it, and enter the desired text.

4. Optional: To enter a description of the newly created destination, click on , and enter the desired text.

5. • To activate the flow of data into the database destination now, click .

   • To activate the flow of data into the database destination later, click in the top right corner of the screen.

     1. When the destination is ready to be activated, find the flow in the All Data Flows screen, and click on the destination.

        

     2. Click on the menu that appears to activate the flow of data to the database destination.