Skip to main content

Dremio Destination

Nexla's bi-directional connectors allow data to flow both to and from any location, making it simple to create a FlexFlow data flow that sends data to a Dremio destination.
dremio_api.png

Dremio

Create a Dremio Destination​

  1. Click the + icon on the Nexset that will be sent to the Dremio destination, and select the Send to Destination option from the menu.

  2. Select the Dremio connector from the list of available destination connectors. Then, select the credential that will be used to connect to the Dremio instance, and click Next; or, create a new Dremio credential for use in this flow.

  3. In Nexla, Dremio destinations can be created using pre-built endpoint templates, which expedite destination setup for common Dremio API write operations. Each template is designed specifically for the corresponding Dremio endpoint, making destination configuration easy and efficient.
    • To configure this destination using a template, follow the instructions in Configure Using a Template.

    Dremio destinations can also be configured manually, allowing you to send data to Dremio API endpoints not included in the pre-built templates or apply further customizations to exactly suit your needs.
    • To configure this destination manually, follow the instructions in Configure Manually.

Configure Using a Template​

Nexla provides pre-built templates that can be used to rapidly configure destinations to send data to common Dremio API endpoints. Each template is designed specifically for the corresponding Dremio endpoint, making destination setup easy and efficient.

  • To configure this destination using a template, select the endpoint to which data will be sent from the Endpoint pulldown menu. Then, click on the template in the list below to expand it, and follow the instructions to configure additional endpoint settings.

    Create Catalog Entity

    Creates a new catalog entity in Dremio, such as a space, folder, virtual dataset (view), or a new registered source. Use this endpoint when you need to programmatically provision new catalog containers or virtual datasets from Nexla as part of an automated data pipeline. For example, you can use this endpoint to create a new virtual dataset (view) in Dremio that references data managed by Nexla.

    • Prepare the Nexset records sent to this destination to conform to the Dremio Catalog entity schema. The body of each API call will be the JSON-serialized record, so each record in the Nexset should represent a complete, valid Dremio catalog entity object. The required fields vary by entity type:

      • Space: Requires entityType set to space and a name field.
      • Folder: Requires entityType set to folder and a path array representing its location in the catalog.
      • Virtual Dataset (View): Requires entityType set to dataset, type set to VIRTUAL_DATASET, a path array, and a virtualDataset object containing the SQL definition.
      • Source: Requires entityType set to source, a name, and a config object specific to the source type (e.g., S3, JDBC).
    • No additional Nexla-side configuration parameters are required beyond selecting this endpoint template. Nexla will POST each record as a JSON body to the Dremio Catalog API at /api/v3/catalog.

    Additional reference information, including detailed entity schema definitions, is available in the Dremio Catalog API documentation.

    Update Catalog Entity

    Updates an existing catalog entity in Dremio by its opaque catalog ID. Use this endpoint to programmatically update source metadata, modify virtual dataset SQL definitions, or refresh configuration details of existing catalog objects. This is useful for workflows that need to keep Dremio catalog entries synchronized with upstream changes managed in Nexla.

    • Enter the opaque catalog ID of the entity to update in the Catalog Entity ID field. This ID is a system-generated string that uniquely identifies the catalog object in Dremio. It can be obtained from the Get Catalog Entity by ID or Get Catalog Entity by Path source endpoints.

    • The Nexset records sent to this destination should each contain the complete updated representation of the catalog entity as a JSON object. The Dremio Catalog Update API uses a full-replace (PUT) model, so all required fields must be present in the body:

      • Include the entity's id, entityType, tag (optimistic concurrency version tag), and all other fields that should be preserved on the entity.
      • Only the fields you include in the body will be retained. Omitting a field may result in it being cleared on the entity.

    The tag field (an optimistic concurrency control token) is required for update operations. Retrieve the current tag value from the entity's existing catalog record before updating. Additional reference information is available in the Dremio Catalog API documentation.

    Delete Catalog Entity

    Deletes a catalog entity from Dremio by its opaque catalog ID. Supported entity types include spaces, folders, virtual datasets (views), physical datasets, and registered sources. Use this endpoint to programmatically remove catalog objects as part of automated data lifecycle management or cleanup pipelines.

    • Enter the opaque catalog ID of the entity to delete in the Catalog Entity ID field. This ID can be obtained from the Get Catalog Entity by ID or Get Catalog Entity by Path source endpoints.
    • The delete operation sends an empty request body to the Dremio Catalog API. Ensure the Catalog Entity ID is correct before activating this destination, as deleted catalog entities cannot be automatically recovered.

    :::warning Important Deleting a space, folder, or source will also delete all catalog entities contained within it. Verify the correct entity ID is configured before activating this destination in a production pipeline. :::

    Additional reference information is available in the Dremio Catalog API documentation.

    Submit SQL Query

    Submits a SQL query to Dremio for asynchronous execution and returns a job ID that can be used to poll for results. Use this endpoint to execute DDL or DML statements, trigger transformations in Dremio views, or run ad-hoc queries as part of a Nexla data pipeline. This is particularly useful for use cases such as refreshing a Dremio Reflection after upstream data has been updated in Nexla.

    • Each Nexset record sent to this endpoint should be a JSON object with the following fields:

      • sql (required): The SQL statement to execute, for example SELECT * FROM mySpace.myView LIMIT 100 or ALTER TABLE mySource.myTable REFRESH METADATA.
      • context (optional): An array of strings representing the default catalog path context for the query, for example ["mySpace", "myFolder"]. Setting a context allows you to write unqualified table names in your SQL.
      • references (optional): A map of source names to version references for querying Iceberg table snapshots at a specific version.
    • The API response includes a id field containing the job ID. Use the Get Job Status and Get Job Results source endpoints to monitor execution and retrieve results after the job completes.

    Additional reference information, including query context and versioning options, is available in the Dremio SQL API documentation.

    Create Reflection

    Creates a new Reflection in Dremio to accelerate queries on a dataset. Reflections are Dremio's query acceleration mechanism—pre-computed, Apache Arrow-formatted materializations of datasets or aggregations that dramatically speed up query execution. Use this endpoint to programmatically provision new raw Reflections (full dataset materializations) or aggregation Reflections (pre-computed aggregates) as part of a data pipeline.

    • Each Nexset record sent to this endpoint should be a JSON object conforming to the Dremio Reflection schema. Key fields include:

      • type (required): Either RAW (for full dataset materialization) or AGGREGATION (for pre-computed aggregates).
      • datasetId (required): The opaque catalog ID of the dataset this Reflection will accelerate.
      • name (required): A descriptive name for the Reflection.
      • displayFields (for RAW Reflections): An array of field name objects specifying which columns to include in the materialization.
      • dimensionFields, measureFields (for AGGREGATION Reflections): Arrays specifying grouping dimensions and aggregate measures.
      • partitionFields, sortFields (optional): Arrays for controlling partitioning and sort order of the materialization.
    • After a Reflection is created, Dremio will begin materializing it in the background. The Reflection will not be used to accelerate queries until it has been fully materialized.

    Additional reference information, including the full Reflection schema, is available in the Dremio Reflections API documentation.

    Cancel Job

    Cancels a running SQL job by its job ID. Use this endpoint to programmatically terminate long-running queries in Dremio, for example as part of a pipeline that detects stalled or timed-out jobs and cleans them up.

    • Enter the ID of the SQL job to cancel in the Job ID field. Job IDs are returned by the Submit SQL Query endpoint when a query is submitted to Dremio, or can be retrieved from the Dremio console Job History view.
    • The cancel operation sends an empty request body to the Dremio Job Cancel API. Only jobs that are currently in a RUNNING or QUEUED state can be canceled; jobs that have already COMPLETED, FAILED, or been CANCELED will not be affected.

    Additional reference information is available in the Dremio Job API documentation.

Configure Manually​

Dremio destinations can be manually configured to send data to any valid Dremio API endpoint.

Using manual configuration, you can also configure Nexla to automatically send the response received from the Dremio API after each call to a new Nexla webhook data source.

API Method​

  1. To manually configure this destination, select the Advanced tab at the top of the configuration screen.

  2. Select the API method that will be used for calls to the Dremio API from the Method pulldown menu. Common methods for Dremio write operations include:

    • POST: For creating new catalog entities, submitting SQL queries, creating Reflections, or canceling jobs
    • PUT: For updating existing catalog entities (full replacement)
    • DELETE: For deleting catalog entities

Data Format​

  1. Select the format in which the Nexset data will be sent to the Dremio API from the Content Format pulldown menu. Dremio's REST API v3 accepts and returns JSON for all write operations. Nexla will automatically convert the data to the selected format for each API call.

API Endpoint URL​

  1. Enter the URL of the Dremio API endpoint to which you want to send the Nexset data in the URL field. All Dremio REST API v3 endpoints follow the pattern {base_url}/api/v3/{resource_path}. For update or delete operations, include the entity or job ID at the end of the URL.

    Common endpoint URL patterns for destination operations:

    • Create catalog entity: {base_url}/api/v3/catalog
    • Update catalog entity: {base_url}/api/v3/catalog/{'{id}'}
    • Delete catalog entity: {base_url}/api/v3/catalog/{'{id}'}
    • Submit SQL query: {base_url}/api/v3/sql
    • Create Reflection: {base_url}/api/v3/reflection
    • Cancel job: {base_url}/api/v3/job/{'{id}'}/cancel

The base URL for your Dremio instance is configured in your Dremio credential. Nexla automatically includes the Authorization: Bearer {token} header in all API calls using the Personal Access Token from your saved credential.

Request Headers​

Optional

  • If Nexla should include any additional request headers in API calls to this destination, enter the headers & corresponding values as comma-separated pairs in the Request Headers field (e.g., header1:value1,header2:value2).

    You do not need to include the Authorization or Content-Type headers here—they are automatically added by Nexla using your saved Dremio credential and the selected Content Format.

Exclude Attributes from the Call​

Optional

  • If any record attributes in the Nexset should be omitted when sending data to this Dremio destination, select the attributes from the Exclude Attributes pulldown menu.

  • Any number of attributes can be selected for exclusion, and all excluded attributes will be shown in the field. To remove an attribute from the list, click the X icon next to the attribute name.

Record Batching​

Optional

  1. If records should be sent to this destination in batched API calls, check the box next to Would you like to batch your records together? to enable record batching.

  2. Enter the maximum number of records that should be batched together in a single API call in the Batch Size field. By default, this value is set to 100.

  3. Select the algorithm that will be used to group records into batches from the Grouping Algorithm pulldown menu. The sample request shown in the panel on the right will be updated to reflect the current batching settings. Some algorithms require additional settings—click on an algorithm listed below to view instructions for configuring these settings.

Property Inside JSON Object

  1. Enter the name of the JSON property that should contain the batched records in the Property Name field.
  2. If any additional properties should be included in the request, enter the properties in the Other Props field in JSON format.

Code

  1. Enter the code that will be used to create the batched request in the code editor below the Grouping Algorithm field.

Response Webhook​

Optional

Nexla can automatically send the response received from the Dremio API after each call to a new Nexla webhook data source. This option allows you to capture the Dremio API response for each operation—such as the job ID returned when submitting a SQL query—and route it into a follow-on Nexla flow for further processing.

  • To enable this option, check the box next to Would you like to process the API response as a Nexla Webhook source?.

Sample Request Payload​

Sample request payloads containing a portion of the Nexset data that will be sent to the Dremio API endpoint based on the current settings are shown in the Sample Payload panel on the right. These samples can be referenced to ensure that the destination and request settings are correctly configured.

  • Click on a sample request payload to expand it and view the complete payload content.
  • Sample payloads are automatically updated with each setting change, making it easy to verify that changes achieve the desired effect.

Endpoint Testing (Manual Configuration)​

After all endpoint settings have been configured, Nexla can send a test payload to the Dremio API to ensure that the destination is configured correctly.

  1. To send a test payload, select the Test button at the top of the Sample Payload panel, and click on a listed sample payload to expand it.

  2. If any modifications to the sample payload are needed, make the necessary changes directly within the sample window.

  3. Click the Send Test Data button at the top of a sample payload to send the test payload to the Dremio API using the current settings.

Save & Activate the Destination​

  • Once all endpoint settings have been configured, click the Done button in the upper right corner of the screen to save and create the destination. To begin sending data to the configured Dremio endpoint, open the destination resource menu, and select Activate.

    The Nexset data will not be sent to the Dremio API until the destination is activated. Destinations can be activated immediately or at a later time, providing full control over data movement.