FlexFlow Data Flows

Nexla’s FlexFlow data flow type is recommended for use in most workflows. This flow type uses the Kafka engine to facilitate seamless high-throughput movement of data from any source to any destination.

1. FlexFlow Data Flows

FlexFlow data flows are highly flexible, allowing customization to fit a wide variety of use cases. Users can create transformations of any level of complexity using Nexla’s built-in transforms or custom transform functions, apply iterative data transformations, create branched flows that send data to multiple destinations, and much more with this flow type.

These data flows can also be edited at any time, and intermediate data products can be sent to destinations as well as shared with other users in the organization.

FlexFlow data flows are the best option for a wide variety of use cases and allow users to take full advantage of the powerful Nexla platform.

2. Create a New FlexFlow Data Flow

The FlexFlow data flow type can be used to create many different kinds of data flows. Basic instructions for setting up a FlexFlow data flow are provided in the following sections.

info

Task-specific instructions for most workflows can be found in other User Guides on Nexla Docs.

2.1 Create the Data Source

1. After logging into Nexla, navigate to the Integrate section by selecting from the platform menu on the left side of the screen.

2. Click at the top of the Integrate toolbar on the left.

3. Select FlexFlow from the list of flow types, and click to proceed to data source creation.

   Learning About Flow Types

   In the Select Flow Type screen, click on a data flow type to view more information about it in the panel on the right, including a brief introductory video.
   Data flows of each type can also be created by clicking the button at the top of this panel.

4. In the data source set-up screen, select a connector tile that matches the source type from which data will be ingested in this flow. Once a connector is selected, the platform will automatically advance to the next setup step.

   Locating Connector Tiles

   Connector tiles can be located by scrolling through the full list or by typing the connector name into the search bar at the top of the list.

   To view only connectors for data sources of a certain type, select a connector category from the menu on the left.

   
   Changing the Selected Source Type

   To change the selected connector type after advancing to a later step in the flow setup process, use the menu at the top of the screen to navigate back to the Select step.

   
   
   

5. In the Authenticate screen, follow the instructions below to create or select the credential that will be used to connect to the data source.

   To create a new credential:

   1. Select the Add Credential tile.

      

   2. Enter and/or select the required information in the Add New Credential pop-up.

      Adding New Credentials

      Credential requirements vary depending on the data source type, with some sources requiring URLs, access keys, and other parameters that must be obtained from the source application/location in addition to account information.

      Detailed information about credential creation for specific sources can be found on the Connectors page.

      
      
      

   3. Once all of the required information has been entered, click at the bottom of the pop-up to save the new credential, and proceed to Step 6.

      
      

   To use a previously added or shared credential:

   1. Select the credential from the list.

      

   2. Click in the upper right corner of the screen, and proceed to Step 6.

      
      

6. Select and configure the source location from which data should be read, including any desired advanced settings (i.e., the frequency at which data should be fetched, data selection settings, etc.).

   Configuring Data Sources

   The data source selection and/or configuration procedure varies according to the source type. Detailed instructions for each data source type can be found on the Connectors page.

   
   

   Example Data Source Configuration Screen (Amazon S3)

   

7. Click in the upper right corner of the screen to save and create the data source. Nexla will now automatically scan the source for data and organize that data into a Nexset, which is a logical data product that is immediately ready to be transformed or sent to any destination.

2.2 Transform the Detected Nexset

In FlexFlow data flows, Nexsets can be transformed using any of the Nexset rule types available in the Nexset Designer, making it easy to transform data to suit any downstream needs. In these flows, multiple transformations can be applied to a single Nexset, and transformations can be applied sequentially.

Transforming Nexsets

To learn more about transforming Nexsets and using the Nexset Designer, see Nexset Designer Overview and the other articles in the Transformations section.

1. Click the icon on a Nexset to open it in the Nexset Designer.

2. Create and apply Nexset Rules to make any necessary transformations to the data in the Nexset.

   Applying Transforms to Nexsets

   The Nexset Designer Overview article provides in-depth information about and instructions for transforming data in the Nexset Designer.

   
   

3. Once the transformation is complete, click in the upper right corner of the screen to save the transformation and close the Nexset Designer.

   
   

2.3 Send the Nexset to a Destination

1. Click the icon on a Nexset to begin sending the data that it contains to a destination.

   

2. In the Send Nexset to Destination screen, select a connector tile that matches the destination type to which data will be sent in this flow. Once a connector is selected, the platform will automatically advance to the next setup step.

   
   Changing the Selected Destination Type

   To change the selected connector type after advancing to a later step in the flow setup process, use the menu at the top of the screen to navigate back to the Select step.

   
   
   

3. Add or select the credential that should be used to connect to the destination by following the same instructions shown in Section 3.1, Step 5.

4. Select and configure the destination to which data will be written, including any desired advanced settings (for example, data formatting options).

   Configuring Data Destinations

   The data destination selection and/or configuration procedure varies according to the destination type. Detailed instructions for each data destination type can be found on the Connectors page.

   
   

   Example Data Destination Configuration Screen (FTP)

   

5. Click in the upper right corner of the screen to save and create the destination.

   Important: Data Movement

   Data will not begin to flow into the destination until it is activated, as shown in Step 6 below.

6. Once created, the destination must be activated to begin the flow of data into the destination. To activate the destination, click the icon on the destination, and select from the dropdown menu.