Skip to main content

Weaviate Destination

Nexla's bi-directional connectors allow data to flow both to and from any location, making it simple to create a FlexFlow, DB-CDC, Spark ETL, or Replication data flow that sends data to a Weaviate location.
weaviate_api.png

Weaviate API

Create a Weaviate Destination​

  1. Click the + icon on the Nexset that will be sent to the Weaviate destination, and select the Send to Destination option from the menu.

  2. Select the Weaviate connector from the list of available destination connectors. Then, select the credential that will be used to connect to the Weaviate instance, and click Next; or, create a new Weaviate credential for use in this flow.

  3. In Nexla, Weaviate destinations can be created using pre-built endpoint templates, which expedite destination setup for common Weaviate endpoints. Each template is designed specifically for the corresponding Weaviate endpoint, making destination configuration easy and efficient.
    • To configure this destination using a template, follow the instructions in Configure Using a Template.

Weaviate destinations can also be configured manually, allowing you to send data to Weaviate endpoints not included in the pre-built templates or apply further customizations to exactly suit your needs.
• To configure this destination manually, follow the instructions in Configure Manually.

Configure Using a Template​

Nexla provides pre-built templates that can be used to rapidly configure destinations to send data to common Weaviate endpoints. Each template is designed specifically for the corresponding Weaviate endpoint, making destination setup easy and efficient.

  • To configure this destination using a template, select the endpoint to which data will be sent from the Endpoint pulldown menu. Then, click on the template in the list below to expand it, and follow the instructions to configure additional endpoint settings.

    Add Objects

    This endpoint template performs batch imports to add multiple data objects and cross-references to your Weaviate instance. Use this template when you need to efficiently import large numbers of objects into Weaviate for vector database operations, AI/ML applications, or semantic search purposes.

    • This endpoint sends data as JSON in the request body to add objects to your Weaviate instance. Each record from your Nexset will be included in a batch import operation. The request body should contain the object data in the format required by the Weaviate API, including all required properties for object creation. The endpoint uses batch mode to efficiently process multiple objects in a single API call.

    This endpoint performs batch imports to add multiple data objects and cross-references to your Weaviate instance. Batch imports are an efficient way to add multiple data objects and cross-references. The endpoint uses batch mode to process multiple objects efficiently, with a maximum of 20 records per batch. The endpoint automatically transforms the data structure to match Weaviate's batch import format, mapping the objects field to your messages.

    The object structure must match the Weaviate API's expected format. Ensure your object data includes all required fields. The endpoint uses QUORUM consistency level for reliable data storage. Optional API keys (OpenAI, Cohere, Hugging Face) configured in your credentials will be automatically included in the request headers if they are set. For detailed information about batch imports, object structures, and available properties, see the Weaviate API documentation.

Configure Manually​

Weaviate destinations can be manually configured to send data to any valid Weaviate API endpoint.

Using manual configuration, you can also configure Nexla to automatically send the response received from the Weaviate API after each call to a new Nexla webhook data source.

API Method​

  1. To manually configure this destination, select the Advanced tab at the top of the configuration screen.

  2. Select the API method that will be used for calls to the Weaviate API from the Method pulldown menu. The most common methods for Weaviate destinations are:

    • POST: For creating new objects or batch imports (most common for Weaviate destinations)
    • PUT: For updating existing objects
    • PATCH: For partial updates to existing objects

Data Format​

  1. Select the format in which the Nexset data will be sent to the Weaviate API from the Content Format pulldown menu. Nexla will automatically convert the data to the selected format for each API call.

The Weaviate APIs typically expect JSON format for most operations. Choose JSON unless you have specific requirements for other formats.

API Endpoint URL​

  1. Enter the URL of the Weaviate API endpoint to which you want to send the Nexset data in the URL field. For batch imports, the URL typically follows the pattern {instance_url}/{version}/batch/objects?consistency_level=QUORUM where {instance_url} is your Weaviate instance URL and {version} is your API version (e.g., v1). For update/upsert operations, include the ID of the object to be updated at the end of the URL.

Ensure the API endpoint URL is correct and accessible with your current credentials. You can test the endpoint using the Test button after configuring the URL. The endpoint requires API key authentication via the Authorization header, which is handled automatically by your credential configuration.

Request Headers​

Optional

  • If Nexla should include any additional request headers in API calls to this destination, enter the headers & corresponding values as comma-separated pairs in the Request Headers field (e.g., header1:value1,header2:value2).

    You do not need to include any headers already present in the credentials. Common headers like Authorization, Content-Type, and Accept are typically handled automatically by Nexla based on your credential configuration. Optional API keys (X-OpenAI-Api-Key, X-Cohere-Api-key, X-HuggingFace-Api-key) configured in your credentials will be automatically included if they are set.

Exclude Attributes from the Call​

Optional

  • If any record attributes in the Nexset should be omitted when sending data to this Weaviate destination, select the attributes from the Exclude Attributes pulldown menu.

  • Any number of attributes can be selected for exclusion, and all excluded attributes will be shown in the field. To remove an attribute from the list, click the X icon next to the attribute name.

Record Batching​

Optional

  1. If records should be sent to this destination in batched API calls, check the box next to Would you like to batch your records together? to enable record batching.

  2. Enter the maximum number of records that should be batched together in a single API call in the Batch Size field. By default, this value is set to 100.

  3. Select the algorithm that will be used to group records into batches from the Grouping Algorithm pulldown menu. The sample request shown in the panel on the right will be updated to reflect the current batching settings. Some algorithms require additional settings—click on an algorithm listed below to view instructions for configuring these settings.

Record batching is particularly useful for Weaviate destinations when you need to process large numbers of objects efficiently. Batching can help reduce API calls and improve performance, especially for vector database operations.

Response Webhook​

Optional

Nexla can automatically send the response received from the Weaviate API after each call to a new Nexla webhook data source. This option allows you to keep track of the status of each API call and any additional information returned after each call.

  • To enable this option, check the box next to Would you like to process the API response as a Nexla Webhook source?.

Response webhooks are particularly useful for Weaviate destinations to track object creation status, batch import confirmations, and any error responses from the Weaviate API.

Sample Request Payload​

Sample request payloads containing a portion of the Nexset data that will be sent to the Weaviate API endpoint based on the current settings are shown in the Sample Payload panel on the right. These samples can be referenced to ensure that the destination and request settings are correctly configured.

  • Click on a sample request payload to expand it and view the complete payload content.
  • Sample payloads are automatically updated with each setting change, making it easy to verify that changes achieve the desired effect.

Endpoint Testing (Manual Configuration)​

After all endpoint settings have been configured, Nexla can send a test payload to the Weaviate API to ensure that the destination is configured correctly.

  1. To send a test payload, select the Test button at the top of the Sample Payload panel, and click on a listed sample payload to expand it.

  2. If any modifications to the sample payload are needed, make the necessary changes directly within the sample window.

  3. Click the Send Test Data button at the top of a sample payload to send the test payload to the Weaviate API using the current settings.

Save & Activate the Destination​

  • Once all endpoint settings have been configured, click the Done button in the upper right corner of the screen to save and create the destination. To send the data to the configured Weaviate endpoint, open the destination resource menu, and select Activate.

    The Nexset data will not be sent to the Weaviate endpoint until the destination is activated. Destinations can be activated immediately or at a later time, providing full control over data movement.