Skip to main content

Vespa Destination

Nexla's bi-directional connectors allow data to flow both to and from any location, making it simple to create a FlexFlow, DB-CDC, Spark ETL, or Replication data flow that sends data to a Vespa location.
vespa_api.png

Vespa

Create a New Data Flow​

  1. To create a new data flow, navigate to the Integrate section, and click the New Data Flow button. Then, select the desired flow type from the list, and click the Create button.

  2. Select the Vespa connector tile from the list of available connectors. Then, select the credential that will be used to connect to the Vespa instance, and click Next; or, create a new Vespa credential for use in this flow.

  3. In Nexla, Vespa destinations can be created using pre-built endpoint templates, which expedite destination setup for common Vespa Document API endpoints. Each template is designed specifically for the corresponding Vespa API endpoint, making destination configuration easy and efficient.
    • To configure this destination using a template, follow the instructions in Configure Using a Template.

    Vespa destinations can also be configured manually, allowing you to send data to Vespa API endpoints not included in the pre-built templates or apply further customizations to exactly suit your needs.
    • To configure this destination manually, follow the instructions in Configure Manually.

Configure Using a Template​

Nexla provides pre-built templates that can be used to rapidly configure destinations to send data to common Vespa Document API endpoints. Each template is designed specifically for the corresponding Vespa API endpoint, making destination setup easy and efficient.

Endpoint Settings​

  • Select the endpoint to which this destination will send data from the Endpoint pulldown menu. Available endpoint templates are listed in the expandable boxes below. Click on an endpoint to see more information about it and how to configure your destination for this endpoint.

    Partial Update Document

    This endpoint updates specific document fields via Vespa's document API. Use this endpoint when you need to update only certain fields of a document without replacing the entire document.

    • Enter the namespace in the Namespace field. The default value is default.
    • Enter the document type in the Document Type field. The default value is msmarco.
    • Enter the document ID in the Document ID field. This is the unique identifier for the document you want to update.
    • This endpoint accepts JSON data in the request body containing only the fields you want to update.

    This endpoint uses POST method for partial document updates. The endpoint accepts JSON data with only the fields to be updated. For more information about partial updates, refer to the Vespa Partial Updates Documentation.

    Conditional Create/Update Document

    This endpoint conditionally upserts documents using Vespa's document API. Use this endpoint when you need to create a document if it doesn't exist, or update it if it does exist.

    • Enter the namespace in the Namespace field. The default value is default.
    • Enter the document type in the Document Type field. The default value is msmarco.
    • Enter the document ID in the Document ID field. This is the unique identifier for the document.
    • This endpoint accepts JSON data in the request body containing the complete document data.

    This endpoint uses POST method with create=true parameter for conditional upserts. The endpoint creates the document if it doesn't exist, or updates it if it does. For more information, refer to the Vespa Document API Guide.

    Create Document (Server-Generated ID)

    This endpoint creates documents with server-generated IDs via Vespa's document API. Use this endpoint when you want Vespa to automatically generate document IDs for new documents.

    • Enter the namespace in the Namespace field. The default value is default.
    • Enter the document type in the Document Type field. The default value is msmarco.
    • Enter the destination cluster in the Destination Cluster field. This specifies which Vespa cluster should store the document.
    • This endpoint accepts JSON data in the request body containing the complete document data.

    This endpoint uses POST method for creating documents with server-generated IDs. The endpoint returns the generated document ID in the response. For more information, refer to the Vespa Document API Reference.

    Create or Replace Document (Client-Specified ID)

    This endpoint upserts documents using Vespa's document management API. Use this endpoint when you need to create or replace a document with a client-specified ID.

    • Enter the namespace in the Namespace field. The default value is default.
    • Enter the document type in the Document Type field. The default value is msmarco.
    • Enter the document ID in the Document ID field. This is the unique identifier you want to assign to the document.
    • Specify the document create option in the Doc Create Option field. Options include true (create if not exists) or false (replace if exists).
    • This endpoint accepts JSON data in the request body containing the complete document data.

    This endpoint uses POST method for upserting documents with client-specified IDs. The endpoint can create new documents or replace existing ones based on the create option. For more information, refer to the Vespa Document API Guide.

Endpoint Testing​

Once the selected endpoint template has been configured, Nexla can send a sample of the data that will be sent according to the current settings. This allows users to verify that the destination is configured correctly before saving.

  • To test the current endpoint configuration, click the Test button to the right of the endpoint selection menu. Sample data will be sent & the response will be displayed in the Endpoint Test Result panel on the right.

  • If the test is not successful or the response is not as expected, review the selected endpoint and associated settings, and make any necessary adjustments. Then, click the Test button again, and check the response to ensure that the destination is configured correctly.

Configure Manually​

Vespa destinations can be manually configured to send data to any valid Vespa API endpoint. Manual configuration provides maximum flexibility for accessing endpoints not covered by pre-built templates or when you need custom API configurations.

With manual configuration, you can also create more complex Vespa destinations, such as destinations that send data to multiple endpoints or destinations that require custom authentication headers or request parameters.

API Method​

  1. To manually configure this destination, select the Advanced tab at the top of the configuration screen.

  2. Select the API method that will be used for calls to the Vespa API from the Method pulldown menu. The most common methods are:

    • POST: For creating or updating documents (most Vespa Document API endpoints use POST)

API Endpoint URL​

  1. Enter the URL of the Vespa API endpoint to which this destination will send data in the Set API URL field. This should be the complete URL including the protocol (https://) and any required path parameters. Vespa Document API endpoints typically follow the pattern {base_url}/document/{api_version}/{namespace}/{document_type}/docid/{document_id}, where path parameters are replaced with actual values.

Ensure the API endpoint URL is correct and accessible with your current credentials. You can test the endpoint using the Test button after configuring the URL. The endpoint URL should use the base URL and API version configured in your credential. Vespa API requires Bearer token authentication, which is automatically included from your credential.

Request Headers​

Optional

  • If Nexla should include any additional request headers in API calls to this destination, enter the headers & corresponding values as comma-separated pairs in the Request Headers field (e.g., header1:value1,header2:value2). Additional headers are often required for API versioning, content type specifications, or custom authentication requirements.

    You do not need to include any headers already present in the credentials. Common headers like Authorization, Content-Type, and Accept are typically handled automatically by Nexla based on your credential configuration. For Vespa, the Authorization header with Bearer token is automatically included from your credential, and Content-Type is typically set to application/json for ingestion endpoints.

Request Body Template​

Optional

  • If the API endpoint requires a specific request body format, you can customize how Nexla formats the data before sending it to the Vespa API by entering a request body template in the Request Body Template field. The template should use {message.json} to include the entire record as JSON, or you can specify individual fields using dot notation (e.g., {message.field_name}).

    For most Vespa ingestion endpoints, the default request body template {message.json} will work correctly, sending the entire record as JSON. You may need to customize the template if the API requires a specific structure or if you need to transform the data before sending. Vespa Document API typically requires JSON documents with fields containing the document data.

Endpoint Testing​

After configuring all settings for the selected endpoint, Nexla can send a sample of the data that will be sent according to the current configuration. This allows users to verify that the destination is configured correctly before saving.

  • To test the current endpoint configuration, click the Test button to the right of the endpoint selection menu. Sample data will be sent & the response will be displayed in the Endpoint Test Result panel on the right.

  • If the test is not successful or the response is not as expected, review the selected endpoint and associated settings, and make any necessary adjustments. Then, click the Test button again, and check the response to ensure that the destination is configured correctly.

Save & Activate the Destination​

  1. Once all of the relevant steps in the above sections have been completed, click the Create button in the upper right corner of the screen to save and create the new Vespa destination. Nexla will now begin sending data to the configured endpoint according to your data flow schedule.