Common Setup for REST API-Based Systems

This article provides general information about interfacing with data systems via REST API-based exchanges in Nexla.

1. REST API-Based Data Systems and Nexla

APIs are one of the most common mechanisms by which software applications allow other applications to access and transfer data. REST APIs use client–server HTTP requests based on a set of common definitions and protocols to provide an efficient method for transferring both real-time and stored data.

Nexla includes both templatized REST API connectors for use with specific applications, which require only user credentials and simple configuration to set up, and a fully customizable generic REST API connector that can be used with any data system that supports REST API-based data exchange.

2. Connecting to a REST API (Data Sources)

With Nexla's connectors, users can quickly and easily add any data system that supports REST API-based data exchange as a data source to begin ingesting, transforming, and moving data in any format. This section provides general instructions and information about connecting to data systems via REST API exchanges.

2.1 Configure the Flow Type & Authentication Settings

1. After logging into Nexla, navigate to the Integrate section by selecting from the platform menu on the left side of the screen.

2. Click at the top of the Integrate toolbar on the left to open the Select Source Type screen.

3. Select the type of data flow to be created from the menu on the left, and click .

   
   Data Flow Types in Nexla

   FlexFlow:
   FlexFlow is a flexible all-in-one data flow type that can be used to create both streaming and real-time data flows that can be used to transform data and/or move data from any source to any destination. This flow type uses the Kafka engine to facilitate seamless high-throughput movement of data from any source to any destination. FlexFlow is the recommended flow type for most workflows.

   DB-CDC:
   DB-CDC data flows use change data capture to replicate tables across databases and/or cloud warehouses. This flow type is streamlined for use in data migration and maintenance workflows, allowing users to quickly create data flows that replicate data across storage locations.

   Spark ETL:
   Spark ETL data flows are designed for rapidly modifying large volumes of data stored in cloud databases or Databricks and moving the transformed data into another cloud storage or Databricks location. This flow type uses the Apache Spark engine and is ideal for large-scale data processing wherein minimizing latency in data movement is a critical need.

   Replication:
   Replication data flows are designed for use in workflows that require high-speed movement of unmodified files between storage systems. This flow type is ideal for use when both retaining file structure and transferring data as quickly as possible are critical.

   
   

4. Select the connector tile matching the REST API-based data system from which data will be read in this flow. Once a connector is selected, the platform will automatically advance to the next setup step.

   
   Changing the Selected Source Type

   To change the selected connector type after advancing to a later step in the flow setup process, use the menu at the top of the screen to navigate back to the Select step.

   
   
   

5. Create a new credential that Nexla should use to connect to the REST API-based system, or select an existing credential; then, click .

   Creating a New Credential

   The information required to create a new credential varies for each REST API-based data system. More detailed instructions are provided for each connector, including pre-configured and generic REST API connectors, in the articles on the REST API-Based Systems page.

   
   

6. For some pre-configured REST API connectors, Nexla provides pre-built templates that can be used to easily fetch data from an endpoint. However, these data sources can also be configured manually to fetch data from any desired endpoint.

   • To configure the REST API-based data source using a template, continue to Section 2.2.
   • To configure the REST API-based data source manually, continue to Section 2.3.
     
     

2.2 Configure the Data Source Using a Template

1. In the Configure screen, select the Template tab.

2. Select the API endpoint from which Nexla will fetch data using the Endpoint pulldown menu.

3. Some endpoint templates require the entry of additional information, such as formatted queries, object IDs, object names, etc. Enter this information in the appropriate fields.

   Query Syntax & Keywords

   Information about query syntax can be found in the documentation for the REST API-based data system.

   Other information, such as object IDs and names, can be found in the associated REST API-based data system.

Example: Salesforce

4. Click to test the endpoint. A preview of the data returned from the REST API-based system according the the current settings will be shown in the Endpoint Test Result pane to the right.

5. Once the data in the test preview is correct, continue to Section 2.4.
   
   

2.3 Manually Configure the Data Source

1. In the Configure screen, select the Advanced tab.

2. Select the API method that should be used for this source from the Method pulldown menu.

3. Enter the URL from which data should be fetched in the Set API URL field.

• The subsections below cover additional advanced settings that can be configured for the API source. These settings are optional. Click on a category in the following list to navigate directly to instructions for the related setting(s).

  • Adding Macros to the API URL
  • Specify Relevant Data
  • Include Additional Metadata
  • Request Headers
  • Paginated APIs
  • Chaining API Calls

• After configuring all desired optional settings, skip to the Test the API Endpoint step step to complete the setup of the API-based data source.

  
  

Adding Macros to the API URL

• To customize and add Nexla macros to the API URL entered in Step 3, click to expand the URL Macros panel and access associated settings.

  Macros in API Calls

  Nexla automatically converts any added macros into values when it executes the associated API call.

  

• To add Nexla macros to the API URL, type { at the appropriate position in the Set API URL field, and a dropdown menu containing all available pre-built functions will appear. Click on a listed macro to add it to the Set API URL field at the current position.

  

  • To specify the date-time format to which Nexla should convert date-time macro values, select the desired format from the Date Format for Date/Time Macro pulldown menu.

    

  • To specify the date or time unit that Nexla should use to perform mathematical operations on date-time macro values, select the desired unit from the Time Unit for Operations pulldown menu.

    

• Column values in any Nexla static or dynamic data lookup can also be added as macros in the API call. To add one or more lookup column value macros, select the lookup that should be used to generate the API macros from the Add Lookups to Supported Macros pulldown menu.

  

  • After selecting the lookup, when { is typed into the Set API URL field, the lookup key columns will appear below the system macros in the dropdown menu and can be added to the API call by clicking on the desired key column(s).

    

Specify Relevant Data

• To configure Nexla to treat only a portion of the received API response as relevant data, enter the path to the relevant data in the Set Path to Data in Response field.

  Path Formatting

  For JSON API responses, enter the JSON path that points to the relevant object/array in the response.

  For XML API responses, use XPATH to specify the path to the relevant data in the response.

  

• Nexla can autogenerate data suggested data paths to be used in this field. To view these suggestions, click to fetch a sample API response from the API URL entered in Step 4. The generated suggestions will be shown in the Suggestions box.

  • Click on a suggested path to add it to the Set Path to Data in Response field.
  

Include Additional Metadata

• When a path is entered in the Set Path to Data in Response field (Step 5), Nexla can also be configured to include additional metadata that falls outside of the entered path when ingesting files from this source. To include additional metadata, enter the path to the relevant metadata in the Path to Metadata in Response field.

  Path Formatting

  For JSON API responses, enter the JSON path that points to the relevant object/array in the response.

  For XML API responses, use XPATH to specify the path to the relevant data in the response.

  

Request Headers

• If Nexla should send additional request headers as part of API calls to this source, enter the headers as comma-separated values in the Request Headers field.

  • Headers present in the credential selected for use with this source do not need to be repeated in this field.
  

Paginated APIs

If the API source is paginated, check the box next to This API is paginated, and configure the settings in Steps i-vii below.

1. Select the iteration type used by the API from the Iteration Type pulldown menu.

   

2. Enter the name of the URL parameter used to set the ID, page number, or token for the selected iteration type in the URL parameter for ID/Token/Page field.

   

3. Enter the value of the URL parameter at which Nexla should begin fetching data. Nexla will begin fetching data from the corresponding page and automatically iterate through all subsequent pages (or until reaching an entered stop value).

   

4. Enter the name of the URL parameter used to set the number of items returned in each page in the URL Param for Items per page field.

   Items per Page

   Although API developers typically set a low default number of items returned per page, it is recommended to set this property to the maximum number allowed by the API for data ingestion.

   

5. Define how many items Nexla should fetch per page of data by entering the corresponding value in the Items Per Page field.

   

6. Optional: To configure Nexla to stop fetching data from the API after receiving a specific page, enter the last page value that should be fetched in the Stop After Page No. field.

   

7. Optional: To configure Nexla to execute multiple parallel requests, enter the number of requests that should be executed in the Parallel Requests field.

   Important: Parallel Request Count

   A high parallelism count can result in flaggin for rate limiting by the API vendor. Thus, typically, the Parallel Requests field should be left as the default value of 1.

   Only modify the value of this setting in specific use cases wherein the source is taking longer to iterate through all pages of data than is acceptable for the workflow.

   

Chaining API Calls

• To chain one or more additional API calls, click in the menu on the left, and repeat the preceding steps to configure each call.

Test the API Endpoint

Once the API source has been configured, click to preview the endpoint response in the Endpoint Test Result panel on the right.

• Ensure that the test preview contains the expected data, and continue to Section 2.4.

2.4 Schedule Data Source Scanning

Nexla can be configured to search for new data from the API source according to a specified frequency and/or time point using the options available under the Scheduling category.

• To specify the frequency at which Nexla will scan the source for new files, select an option from the left-most pulldown menu next to Check for files:.

• To specify the time at which Nexla start scanning the source for new files, select the appropriate hour, minute, and AM/PM options using the pulldown menus on the right next to Check for files:.

2.5 Save and Create the Data Source

1. Once all of the above steps have been completed, click in the upper right corner of the screen to save and create the new Salesforce data source.

   • Nexla will now begin scanning the REST API-based data system for data according to the configured settings and will organize any data that it finds into one or more Nexsets.

3. Sending Data to a REST API (Data Destinations)

Nexla's bi-directional connectors allow data to flow both to and from any location, making it simple to set up up a data flow that sends data to a REST API-based data system. This section provides general instructions and information about sending data to REST API-based data systems, along with an interactive tutorial.

3.1 Configure the Destination Type & Authentication Settings

1. Locate the Nexset that will be sent to the REST API-based data system destination.

   Locating Nexsets

   • To view all accessible Nexsets within their associated data flows, navigate to the Integrate section using the platform menu on the left side of the screen. Then, in the All Data Flows screen, click on a listed data flow to view all detected and transformed Nexsets that it contains.

   • To view a list of all accessible Nexsets, navigate to the Integrate section using the platform menu, and select the Nexsets screen.

     
   
   

2. Click the icon on the Nexset. This will open the Send Nexset to Destination screen.

   Transforming Nexsets

   To transform the Nexset before sending it to a destination, click the icon on the Nexset to open it in the Nexset Designer.

   For a complete guide to using the Nexset Designer to transform Nexsets, see Nexset Designer Overview.

3. In the Send Nexset to Destination screen, select the connector tile matching the REST API-based data system to which data will be sent in this flow. Once a connector is selected, the platform will automatically advance to the next setup step.

   
   Changing the Selected Destination Type

   To change the selected connector type after advancing to a later step in the flow setup process, use the menu at the top of the screen to navigate back to the Select step.

   
   
   

4. Create a new credential that Nexla should use to connect to the REST API-based system, or select an existing credential; then, click .

   Creating a New Credential

   The information required to create a new credential varies for each REST API-based data system. More detailed instructions are provided for each connector, including pre-configured and generic REST API connectors, in the articles on the REST API-Based Systems page.

   
   

5. For some pre-configured REST API connectors, Nexla provides pre-built templates that can be used to easily configure the API destination to suit common use cases. Manual API endpoint However, these data destinations can also be configured manually to send data to any desired endpoint.

   • To configure API destination using a template, follow the instructions in Section 3.2.
   • To configure the API destination manually, follow the instructions in Section 3.3.

3.2 Configure the Destination Using a Template

1. In the Configure screen, select the Template tab.

2. Select the API endpoint to which Nexla will send data from the Endpoint pulldown menu.

   Endpoint Templates

   Templates are created and customized specifically for each REST API-based connector. These templates allow users to easily configure REST API destinations for common endpoints & use cases.

Example: JIRA Connector Endpoint Templates

3. Some endpoint templates require the entry of additional information, such as formatted queries, object IDs, object names, etc. Enter this information in the appropriate fields.

   API Endpoint Parameters

   Formatting information, such as query syntax, can be found in documentation available from the vendor of the destination data system.

   Other information—for example, object IDs and names—can be found within the destination data system interface and/or account settings.

Example: JIRA Connector – Add Comments to Issue

4. Sample request payloads that will be sent to the destination can be viewed in the right-hand panel when the Sample Payload tab is selected. Click on a sample payload to view the complete request payload that will be sent for the corresponding Nexset record. These sample payloads can be reviewed to ensure that the API destination will receive all necessary information in the required format.

5. Once the data in the payload preview is correct, continue to Section 3.4.

3.3 Manually Configure the Destination

1. In the Configure screen, select the Advanced tab.

2. Select the API method that should be used to send data to the API destination from the Method pulldown menu.

   • GET – This method returns a representation of the information contained in the API endpoint. It is not typically used when sending data to an API destination.
   • POST – This method creates new subordinate resources following the appropriate hierarchy used by the API endpoint.
   • PUT – This method updates resources in the API endpoint by entirely replacing the resource content with the information in the Nexset records.
   • PATCH – This method updates resources in the API endpoint by modifying the resource content based on the information in the Nexset records.
   • DELETE – This method removes the targeted resources from the API endpoint.

3. Select the payload format that will be used when sending requests to the API endpoint from the Content Format pulldown menu. Ensure that content in the selected format is accepted by the API endpoint for the request method chosen in the previous step.

4. Enter the URL of the API endpoint to which data will be sent int the URL field.

• Steps 5-7 cover additional advanced settings that can be configured for the API endpoint. Configuring these settings is optional. Click on a category in the following list to navigate directly to instructions for the related setting(s).

  • Exclude Attributes
  • Record Batching
  • Response Webhook

• After configuring all desired optional settings, skip to the Payload Preview step to complete the setup of the API endpoint destinaiton.

  
  

Exclude Attributes

• To exclude one or more record attributes included in the Nexset from the data sent to the API endpoint, select the attribute(s) from the Exclude Attributes pulldown menu. Any number of attributes can be selected for exclusion.

• Once an attribute is selected for exclusion, it will be shown in the Exclude Attributes field.

  • To remove a selected attribute from the excluded list, click the X on that attribute in the Exclude Attributes field.

Record Batching

• To configure Nexla to batch the Nexset records when sending the payload(s) to the API endpoint, check the box next to Would you like to batch your records together?. Then, complete steps 1-2 below.

1. Enter the number of records to be included in each batch in the Batch Size field.

   

2. Nexla can be configured to use any custom logic to batch records—for example, multiple records can be batched as an array or as properties inside a JSON object. Records can also be batched according to any custom-coded logic. Select the batch grouping algorithm that should be used to batch records from the Grouping Algorithm pulldown menu.

   

Response Webhook

For some workflows, users need to capture the response from the API endpoint for additional processing—for example, when posting data to an API to create objects in a SaaS platform, the IDs of the newly created objects may need to be captured and saved in a database for bookkeeping. In these cases, Nexla can create a new webhook source and then automatically capture & send the API response to that webhook source.

• To enable this option, check the box next to Would you like to process the API response as a Nexla Webhook Source?. Nexla will create a new webhook when the destination is saved and attach the webhook to the API destination resource.

Payload Preview

Select the Sample Payload button in the right-hand panel to view sample request payloads that will be sent to the API destination. Click on a sample payload to view the complete request payload that will be sent for the corresponding Nexset record. These sample payloads can be reviewed to ensure that the API destination will receive all necessary information in the required format.

Destination Testing

Sending Nexset data to an API destination requires that settings such as the endpoint URL and request method are correctly configured and that any request formatting requirements are met. When setting up an API destination, Nexla users can send a test request to the API before saving the destination to ensure that the Nexset data will be successfully received.

To learn how to send a test request to the API destination and find tips for understanding and using test results, see the Test the Configuration section in Writing Data to an API Destination.

3.4 Save & Activate the Destination

1. Once all of the above steps have been completed, click in the upper right corner of the screen to save and create the new API endpoint destination.

   Important: Data Movement

   Data will not begin to flow into the destination until it is activated, as shown in Step 2 below.

2. Once created, the destination must be activated to begin the flow of data into the destination. To activate the destination, click the icon on the destination, and select from the dropdown menu.

   Data Movement

   After the destination is activated, data will automatically begin to flow to the API endpoint according to the configured settings.

   For data flows with the webhook option enabled in Section 3.3 Step 7, the webhook destination will also be activated.

3.5 Interactive Tutorial

Use the interactive tutorial below to follow a hands-on demonstration of how to set up a REST API destination within the Nexla UI.