DirectFlow Data Flows

DirectFlow data flows are designed for high-throughput point-to-point data processing wherein minimizing data processing latency is a critical need. This flow type is ideal for use in batch processing with non-streaming data sources & destinations.

1. DirectFlow Data Flows

In DirectFlow data flows, data processing latency is minimized by executing the entire flow within the memory of a single container to achieve the highest possible throughput. These flows can be constructed to ingest data from any non-streaming data source and destination

Similar to FlexFlows, the DirectFlow flow type supports data transformations created with pre-built or custom-coded transform rules to modify ingested data prior to sending it to the destination. However, these flows do not support branching to send data to multiple destinations.

Use Cases: DirectFlow Data Flows

DirectFlow data flows provide the ability to transform ingested data before sending it to a destination, but this flow type does not support branching and cannot be used to send data to multiple destinations. These flows also do not support Kafka, webhook or other realtime data sources/destinations.

For workflows that require sending data to more than one location and/or the inclusion of a realtime data source or destination, the FlexFlow flow type is recommended.

DirectFlow Data Flow

Setting up a DirectFlow data flow in Nexla to maximize data throughput is easy. The following sections provide step-by-step instructions for creating a new DirectFlow data flow, along with information about optional features and settings available for this flow type.

2. Create a New DirectFlow Data Flow & Data Source

1. After logging into Nexla, navigate to the Integrate section by selecting from the platform menu on the left side of the screen.

2. Click at the top of the Integrate toolbar on the left.

3. Select DirectFlow from the list of flow types, and click to proceed to data source creation.

   Learning About Flow Types

   In the Select Flow Type screen, click on a data flow type to view more information about it in the panel on the right, including a brief introductory video.

   Data flows of each type can also be created by clicking the button at the top of this panel.

Configure the Data Source

1. In the Connect screen, select the connector tile matching the desired data source type from the list.

   • DirectFlow data flows do not support streaming data sources, such as webhooks, Kafka sources, and other realtime sources. Only supported connectors are shown in the Connect screen.

   • To create a data flow with a data source type not available in this screen, use the FlexFlow flow type or one of the other flow types listed on the Nexla Data Flow Types page.

2. In the Authenticate screen, select or create the credential that will be used to connect to the data source, and click . Detailed information about credential creation for specific sources can be found on the Connectors page.

3. Enter a name for the data source in the Name field.

4. Optional: Enter a description of the data source in the Description field.

   Resource Descriptions

   Resource descriptions should provide information about the resource purpose, data freshness, etc. that can help the owner and other users efficiently understand and utilize the resource.

5. Configure the data source settings, such as the location from which data will be ingested and the ingestion frequency, according to the selected source type.

   Configuring a Data Source

   For information about configuring each data source type, see the following pages in the Connectors section:

   • REST API-Based Systems
   • File-Based Systems
   • Databases & Data Warehouses

6. After configuring all required settings and any desired additional options, click in the top right corner of the screen to save & activate the data source.

   Once the data source is created, Nexla will automatically scan it for data according to the configured settings. Identified data will be organized into a Nexset, which is a logical data product that is immediately ready to be sent to a destination.

3. Transform the Detected Nexset

DirectFlow data flows support the application of transforms to modify ingested data before sending it to the destination. In the Nexset Designer, ingested data can be modified using Nexla's pre-built transforms or custom code-based transforms.

However, Nexset transformation is optional. To set up a DirectFlow data flow without transforming the detected Nexset, skip to Section 4.

1. To begin constructing a transform, click the icon on the Nexset to open it in the Nexset Designer.

2. Create and apply Nexset Rules to make any necessary transformations to the data in the Nexset.

   Applying Transforms to Nexsets

   The Nexset Designer Overview article provides in-depth information about and instructions for transforming data in the Nexset Designer.

   
   

3. Once the transformation is complete, click in the upper right corner of the screen to save the transformation and close the Nexset Designer.

   
   

4. Create a Destination

1. Click the icon on the Nexset to begin sending the data that it contains to a destination.

2. In the Configure screen, select a connector tile that matches the destination type to which data will be sent in this flow. Once a connector is selected, the platform will automatically advance to the next setup step.

3. In the Authenticate screen, select or create the credential that will be used to connect to the data source, and click . Detailed information about credential creation for specific sources can be found on the Connectors page.

Configure the Destination

1. Enter a name for the destination in the Name field.

2. Optional: Enter a description of the destination in the Description field.

   Resource Descriptions

   Resource descriptions should provide information about the resource purpose, data freshness, etc. that can help the owner and other users efficiently understand and utilize the resource.

3. Configure the location to which data will be sent and any other desired settings according to the selected destination type.

   Configuring a Destination

   For information about configuring each destination type, see the following pages in the Connectors section:

   • REST API-Based Systems
   • File-Based Systems
   • Databases & Data Warehouses

   Example Destination Configuration Screen (Database)

   

Save & Activate the Destination

1. After configuring all necessary settings, click in the upper right corner of the screen to save and create the destination.
   Important: Data Movement

   Data will not begin to flow into the destination until it is activated, as shown in the following step.

2. Once created, the destination must be activated to begin the flow of data into the destination. To activate the destination, click the icon on the destination, and select from the dropdown menu.