Vespa

Vespa is an open-source big data serving engine that enables vector search, filtering, and document operations in real time. The Vespa connector enables you to access Vespa's Document API to store, retrieve, update, and search documents with vector embeddings. This connector is particularly useful for applications that need to perform vector similarity search, build recommendation systems, implement real-time search capabilities, or manage large-scale document stores with advanced querying features.

Power end-to-end data operations for your Vespa API with Nexla. Our bi-directional Vespa connector is purpose-built for Vespa, making it simple to ingest data, sync it across systems, and deliver it anywhere — all with no coding required. Nexla turns API-sourced data into ready-to-use, reusable data products and makes it easy to send data to Vespa or any other destination. With comprehensive monitoring, lineage tracking, and access controls, Nexla keeps your Vespa workflows fast, secure, and fully governed.

Features

Type: API

SourceDestination

• Seamless API Integration: Connect to any endpoint as source or destination without coding, with automatic data product creation
• Visual Composition & Chaining: Build complex integrations using visual templates, chain API calls, and compose workflows with data validation and filtering
• API Proxy: Expose curated slices of your data securely with a secure and customizable API proxy that validates and transforms data on the fly
• Request optimization with intelligent batching, retry, and caching to minimize API calls and costs

Prerequisites

Before creating a Vespa credential, you'll need to obtain an API key from your Vespa deployment. Vespa uses API key authentication for headless communication with Vespa Cloud control plane services, or you may need to configure authentication based on your specific Vespa deployment.

To obtain Vespa API credentials:

1. If you're using Vespa Cloud, log in to your Vespa Cloud account at https://console.vespa-cloud.com.

2. Navigate to your application or tenant settings in the Vespa Cloud console.

3. In the API settings or security section, locate the option to create or manage API keys.

4. Click Create API Key or Generate New Key to create a new API key for your application. You may need to provide a name or description for the API key.

5. Copy the API key immediately after generation, as it may only be displayed once for security purposes. Store it securely, as you'll need it to authenticate API requests.

6. Note your Base URL. This is the root URL for your Vespa API endpoints, including the protocol (https://) and port. For Vespa Cloud, this is typically provided in your application configuration. For self-hosted Vespa, this is your Vespa instance URL.

7. Note your API Version. The default API version is typically v1, but this may vary depending on your Vespa deployment.

Vespa API keys are used as Bearer tokens in the Authorization header for all API requests. The API key is sensitive information and should be kept secure. If you're using a self-hosted Vespa instance, authentication may be configured differently. Consult your Vespa deployment documentation for specific authentication requirements.

For detailed information about Vespa API authentication and API key management, refer to the Vespa Document API Guide.

Authenticate

Credentials required

An authentication method that requires sending a unique secret token with each API request on Vespa AI. (Include the https:// on it.)

Field	Required	Secret	Description
Base URL	Yes	No	Root URL for the Vespa API endpoints. Must include protocol and port.
Vespa API Key	Yes	Yes	Authentication token for headless communication with Vespa Cloud control plane services
Vespa API Version	Yes	Yes	The API Version for Vespa AI (Default: v1)

Create a credential in Nexla

1. After selecting the data source/destination type, click the Add Credential tile to open the Add New Credential overlay.

New Credential Overlay – Vespa

2. Enter a name for the credential in the Credential Name field and a short, meaningful description in the Credential Description field.

3. In the Base URL field, enter the root URL for the Vespa API endpoints. This must include the protocol (https://) and port if applicable. For example, https://your-vespa-instance.com:8080 or your Vespa Cloud application URL.

4. In the Vespa API Key field, enter the API key that you obtained from your Vespa deployment. This is the authentication token for headless communication with Vespa Cloud control plane services or your Vespa instance.

5. In the Vespa API Version field, enter the API version for Vespa API requests. The default value is v1. This field is required and should be kept secure.

   The API key is sensitive information and should be kept secure. If you've lost your API key, you'll need to generate a new one in your Vespa Cloud console or Vespa deployment settings. API keys are used as Bearer tokens in the Authorization header for all API requests.

6. Click the Save button at the bottom of the overlay. The newly added credential will now appear in a tile on the Authenticate screen during data source/destination creation and can be selected for use with a new data source or destination.

Use as a data source

To create a new data flow, navigate to the Integrate section, and click the New Data Flow button. Select the Vespa connector tile, then select the credential that will be used to connect to the Vespa API, and click Next; or, create a new Vespa credential for use in this flow.

Endpoint templates

Nexla provides pre-built templates that can be used to rapidly configure data sources to ingest data from common Vespa API endpoints. Each template is designed specifically for the corresponding Vespa API endpoint, making data source setup easy and efficient. Select the endpoint from which this source will fetch data from the Endpoint pulldown menu. Available endpoint templates are listed in the expandable boxes below.

Retrieve Document by ID

This endpoint retrieves documents by ID using Vespa's document API. Use this endpoint when you need to fetch specific documents, retrieve document details, or access documents by their unique identifiers.

• Enter the namespace in the Namespace field. The default value is default. This identifies the document namespace in Vespa.
• Enter the document type in the Document Type field. The default value is msmarco. This is the document type defined by your Vespa schema.
• Enter the document ID in the Document ID field. This is the unique identifier for the document you want to retrieve.

The Retrieve Document by ID endpoint uses GET requests to retrieve documents from the Vespa Document API. The endpoint returns the complete document data including all fields and metadata. For more information about the Retrieve Document by ID endpoint, refer to the Vespa Document API Reference.

Vector Search with Custom Body

This endpoint enables vector-based semantic searches using custom JSON queries via Vespa's search API. Use this endpoint when you need to perform vector similarity search, implement semantic search, or query documents using vector embeddings.

• Enter the custom query body in JSON format in the Body field. This should include your search query with vector search parameters, filters, and ranking expressions.

The Vector Search with Custom Body endpoint uses POST requests to execute custom search queries against the Vespa search API. The endpoint supports vector similarity search, filtering, and advanced ranking. For more information about vector search, refer to the Vespa Ranking Documentation.

Once the selected endpoint template has been configured, click the Test button to the right of the endpoint selection menu to retrieve a sample of the data that will be fetched. Sample data will be displayed in the Endpoint Test Result panel on the right, allowing you to verify that the source is configured correctly before saving.

Manual configuration

Vespa data sources can also be manually configured to ingest data from any valid Vespa API endpoint, including endpoints not covered by the pre-built templates, chained API calls, or custom request parameters. Select the Advanced tab at the top of the configuration screen, and follow the instructions in Connect to Any API to configure the API method, endpoint URL, date/time and lookup macros, path to data, metadata, and request headers.

Vespa Document API endpoints typically follow the pattern {base_url}/document/{api_version}/{namespace}/{document_type}/docid/{document_id} and use the GET method, while the Search API uses {base_url}/search/ with the POST method. For Path to Data, common paths include $ for the entire document, $.root.children[*] for search results, or $.fields for document fields. The Authorization header with Bearer token is automatically included from your credential. For POST requests to the Search API, the Request Body should be JSON containing a yql field for the query string, hits for the number of results, ranking for ranking expressions, and queryProfile for query profiles.

Once all of the relevant settings have been configured, click the Create button in the upper right corner of the screen to save and create the new Vespa data source. Nexla will now begin ingesting data from the configured endpoint and will organize any data that it finds into one or more Nexsets.

Use as a destination

Click the + icon on the Nexset that will be sent to the Vespa destination, and select the Send to Destination option from the menu. Select the Vespa connector from the list of available destination connectors, then select the credential that will be used to connect to the Vespa instance, and click Next; or, create a new Vespa credential for use in this flow.

Endpoint templates

Nexla provides pre-built templates that can be used to rapidly configure destinations to send data to common Vespa Document API endpoints. Select the endpoint to which this destination will send data from the Endpoint pulldown menu. Then, click on the template in the list below to expand it, and follow the instructions to configure additional endpoint settings.

Partial Update Document

This endpoint updates specific document fields via Vespa's document API. Use this endpoint when you need to update only certain fields of a document without replacing the entire document.

• Enter the namespace in the Namespace field. The default value is default.
• Enter the document type in the Document Type field. The default value is msmarco.
• Enter the document ID in the Document ID field. This is the unique identifier for the document you want to update.
• This endpoint accepts JSON data in the request body containing only the fields you want to update.

This endpoint uses POST method for partial document updates. The endpoint accepts JSON data with only the fields to be updated. For more information about partial updates, refer to the Vespa Partial Updates Documentation.

Conditional Create/Update Document

This endpoint conditionally upserts documents using Vespa's document API. Use this endpoint when you need to create a document if it doesn't exist, or update it if it does exist.

• Enter the namespace in the Namespace field. The default value is default.
• Enter the document type in the Document Type field. The default value is msmarco.
• Enter the document ID in the Document ID field. This is the unique identifier for the document.
• This endpoint accepts JSON data in the request body containing the complete document data.

This endpoint uses POST method with create=true parameter for conditional upserts. The endpoint creates the document if it doesn't exist, or updates it if it does. For more information, refer to the Vespa Document API Guide.

Create Document (Server-Generated ID)

This endpoint creates documents with server-generated IDs via Vespa's document API. Use this endpoint when you want Vespa to automatically generate document IDs for new documents.

• Enter the namespace in the Namespace field. The default value is default.
• Enter the document type in the Document Type field. The default value is msmarco.
• Enter the destination cluster in the Destination Cluster field. This specifies which Vespa cluster should store the document.
• This endpoint accepts JSON data in the request body containing the complete document data.

This endpoint uses POST method for creating documents with server-generated IDs. The endpoint returns the generated document ID in the response. For more information, refer to the Vespa Document API Reference.

Create or Replace Document (Client-Specified ID)

This endpoint upserts documents using Vespa's document management API. Use this endpoint when you need to create or replace a document with a client-specified ID.

• Enter the namespace in the Namespace field. The default value is default.
• Enter the document type in the Document Type field. The default value is msmarco.
• Enter the document ID in the Document ID field. This is the unique identifier you want to assign to the document.
• Specify the document create option in the Doc Create Option field. Options include true (create if not exists) or false (replace if exists).
• This endpoint accepts JSON data in the request body containing the complete document data.

This endpoint uses POST method for upserting documents with client-specified IDs. The endpoint can create new documents or replace existing ones based on the create option. For more information, refer to the Vespa Document API Guide.

Once the selected endpoint template has been configured, click the Test button to the right of the endpoint selection menu to retrieve a sample of the data that will be sent. Sample data will be displayed in the Endpoint Test Result panel on the right, allowing you to verify that the destination is configured correctly before saving.

Manual configuration

Vespa destinations can also be manually configured to send data to any valid Vespa API endpoint. Select the Advanced tab at the top of the configuration screen, and follow the instructions in Connect to Any API to configure the API method, data format, endpoint URL, request headers, attribute exclusions, record batching, and response webhooks.

Vespa Document API endpoints typically follow the pattern {base_url}/document/{api_version}/{namespace}/{document_type}/docid/{document_id} and use the POST method for most Document API operations. The Authorization header with Bearer token is automatically included from your credential, and Content-Type is typically set to application/json for ingestion endpoints. For most Vespa ingestion endpoints, the default Request Body Template of {message.json} sends the entire record as JSON; Vespa's Document API expects JSON documents with a fields object containing the document data, so customize the template with dot notation (e.g., {message.field_name}) if you need a different structure.

Save & activate

Once all of the relevant steps in the above sections have been completed, click the Create button in the upper right corner of the screen to save and create the new Vespa destination. Nexla will now begin sending data to the configured endpoint according to your data flow schedule.