Elasticsearch

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. The Elasticsearch connector enables you to efficiently manage and manipulate document records using Elasticsearch's bulk document operations API. You can index documents, perform bulk updates, and manage data in your Elasticsearch clusters.

Power end-to-end data operations for your Elasticsearch API with Nexla. Our bi-directional Elasticsearch connector is purpose-built for Elasticsearch, making it simple to ingest data, sync it across systems, and deliver it anywhere — all with no coding required. Nexla turns API-sourced data into ready-to-use, reusable data products and makes it easy to send data to Elasticsearch or any other destination. With comprehensive monitoring, lineage tracking, and access controls, Nexla keeps your Elasticsearch workflows fast, secure, and fully governed.

Features

Type: API

SourceDestination

• Seamless API Integration: Connect to any endpoint as source or destination without coding, with automatic data product creation
• Visual Composition & Chaining: Build complex integrations using visual templates, chain API calls, and compose workflows with data validation and filtering
• API Proxy: Expose curated slices of your data securely with a secure and customizable API proxy that validates and transforms data on the fly
• Request optimization with intelligent batching, retry, and caching to minimize API calls and costs

Prerequisites

Before creating an Elasticsearch credential, you'll need to obtain authentication credentials for your Elasticsearch cluster. Elasticsearch supports multiple authentication methods, including basic authentication (username/password) and API key authentication.

Elasticsearch Authentication Setup

To obtain the required authentication credentials for Elasticsearch:

1. Access Your Elasticsearch Cluster: Connect to your Elasticsearch cluster. This could be:

   • A self-managed Elasticsearch cluster
   • Elasticsearch Cloud (Elastic Cloud)
   • An Elasticsearch service provided by a cloud provider

2. Determine Authentication Method: Elasticsearch supports several authentication methods:

   • Basic Authentication: Username and password credentials
   • API Key Authentication: API keys generated from the Elasticsearch cluster
   • Cloud ID Authentication: For Elasticsearch Cloud deployments

3. For Basic Authentication:

   • Use your Elasticsearch cluster username and password. These are typically configured during cluster setup or can be managed through the Elasticsearch security settings.
   • For Elasticsearch Cloud, you can find credentials in your deployment settings or use the default elastic user credentials.

4. For API Key Authentication:

   • Generate an API key from your Elasticsearch cluster. This can typically be done through:
     • The Elasticsearch API: POST /_security/api_key
     • Elasticsearch Cloud dashboard: Navigate to your deployment settings and look for API keys
     • Kibana: If you have Kibana access, you can generate API keys through the security settings
   • Store the generated API key securely, as it provides access to your cluster.

5. Get Base URL and Index Name:

   • Note your Elasticsearch cluster base URL (e.g., https://your-cluster.es.amazonaws.com or https://your-deployment.es.cloud.io)
   • Identify the index name you'll be working with (e.g., my-index)

Elasticsearch credentials are sensitive and provide access to your cluster and data. Keep these credentials secure and never share them publicly or commit them to version control systems. For Elasticsearch Cloud, you can find authentication details in your deployment settings. For self-managed clusters, credentials are typically configured during cluster setup.

API Configuration Details

The Elasticsearch API requires the following configuration:

• Base URL: The base URL for your Elasticsearch cluster (e.g., https://your-cluster.es.amazonaws.com or https://your-deployment.es.cloud.io)
• Index Name: The name of the Elasticsearch index you'll be working with
• Authentication: Either basic authentication (username/password) or API key authentication

All requests to the Elasticsearch API must include authentication headers. For basic authentication, credentials are sent in the Authorization header. The API accepts JSON and NDJSON (newline-delimited JSON) formats for bulk operations.

For complete information about Elasticsearch authentication and API access, see the Elasticsearch API Documentation.

Authenticate

Credentials required

A standard authentication method that requires sending encoded username and password credentials along with the requests for Elasticsearch

Field	Required	Secret	Description
Username Or API Key	Yes	No	Your username or personal API Key for Elasticsearch
Password	Yes	Yes	Your password for Elasticsearch
Base URL	Yes	No	The Elasticsearch server URL for API operations. Format: https://host:port. Used in bulk indexing, search, and document management.
Authentication Credentials	Yes	No	Authentication credentials for API access. String format (username:password). Required for secure operations.
Index Name	Yes	No	Specifies the Elasticsearch index name for document operations. Required string parameter for bulk indexing, search, and CRUD operations.

Create a credential in Nexla

1. After selecting the data source/destination type, click the Add Credential tile to open the Add New Credential overlay.

New Credential Overlay – Elasticsearch

2. Enter a name for the credential in the Credential Name field and a short, meaningful description in the Credential Description field.

3. Enter your Elasticsearch username or API key in the Username Or API Key field. This is used for basic authentication with your Elasticsearch cluster. If you're using API key authentication, enter the API key here.

4. Enter your Elasticsearch password in the Password field. This is required for basic authentication. If you're using API key authentication, you may leave this empty or enter a placeholder value.

5. Enter the base URL for your Elasticsearch cluster in the Base URL field. This should be the complete URL including the protocol (https://) and domain (e.g., https://your-cluster.es.amazonaws.com or https://your-deployment.es.cloud.io).

6. Enter your username and password together in the Authentication Credentials field, in the format username:password. This value is required and is used together with the fields above to construct the encoded authorization used by Elasticsearch's bulk operations API.

7. Enter the name of the Elasticsearch index you'll be working with in the Index Name field (e.g., my-index). This index will be used for data operations.

The authentication credentials are sensitive information that should be kept secure. Nexla will store this credential securely and use it only for API authentication purposes. For Elasticsearch Cloud deployments, you can find your base URL and credentials in your deployment settings. For self-managed clusters, use the cluster URL and credentials configured during setup.

8. Click the Save button at the bottom of the overlay. The newly added credential will now appear in a tile on the Authenticate screen during data source/destination creation and can be selected for use with a new data source or destination.

Use as a data source

To create a new data flow, navigate to the Integrate section, and click the New Data Flow button. Select the Elasticsearch connector tile, then select the credential that will be used to connect to the Elasticsearch instance, and click Next; or, create a new Elasticsearch credential for use in this flow.

Endpoint templates

Nexla provides pre-built templates that can be used to rapidly configure data sources to ingest data from common Elasticsearch endpoints. Select the endpoint from which this source will fetch data from the Endpoint pulldown menu. Available endpoint templates are listed in the expandable boxes below.

Search Documents

This endpoint performs a search query on an Elasticsearch index to retrieve matching documents. Use this endpoint when you need to search for documents, filter results, or extract data based on specific criteria.

• Enter the search query in the Query field. This should be a valid Elasticsearch query in JSON format. You can use match queries, term queries, range queries, and other Elasticsearch query types. Leave empty to retrieve all documents (up to API limits).
• Optionally, enter the maximum number of results to return in the Size field. The default is 10. This controls how many documents are returned per request.
• Optionally, enter the starting offset for pagination in the From field. The default is 0. This allows you to paginate through large result sets.
• Enter a schedule in the Schedule field to specify when this data source should run. The schedule uses cron expression format (e.g., 0 6 * * * for daily at 6 AM).
• The endpoint uses the GET method to /your-index/_search with the query in the request body or as query parameters.

Elasticsearch search queries support a wide range of query types including match, term, range, bool, and more. The query syntax follows Elasticsearch's Query DSL. For complete information about Elasticsearch search queries, see the Elasticsearch Search API Documentation.

Get Document by ID

This endpoint retrieves a specific document from an Elasticsearch index by its document ID. Use this endpoint when you need to retrieve a specific document, fetch document metadata, or get a single document by its unique identifier.

• Enter the Document ID in the Document ID field. This should be the exact ID of the document you want to retrieve from the Elasticsearch index.
• Enter a schedule in the Schedule field to specify when this data source should run. The schedule uses cron expression format.
• The endpoint uses the GET method to /your-index/_doc/{document_id} to retrieve the specified document.

Document IDs in Elasticsearch are unique identifiers for each document in an index. You can find document IDs from search results or when indexing documents. For complete information about getting documents by ID, see the Elasticsearch Get API Documentation.

Once the selected endpoint template has been configured, click the Test button to the right of the endpoint selection menu to retrieve a sample of the data that will be fetched. Sample data will be displayed in the Endpoint Test Result panel on the right, allowing you to verify that the source is configured correctly before saving.

Manual configuration

Elasticsearch data sources can also be manually configured to ingest data from any valid Elasticsearch API endpoint, including endpoints not covered by the pre-built templates, chained API calls, or custom request parameters. Select the Advanced tab at the top of the configuration screen, and follow the instructions in Connect to Any API to configure the API method, endpoint URL, date/time and lookup macros, path to data, metadata, and request headers.

Elasticsearch API URLs typically follow the format https://your-cluster.es.amazonaws.com/your-index/_search for search operations, or https://your-cluster.es.amazonaws.com/your-index/_doc/{document_id} for getting specific documents. For search responses, set the Response Data Path to $.hits.hits[*] to extract individual documents from the hits array; for single document responses, use $ (the _source field contains the actual document data).

Once all of the relevant settings have been configured, click the Create button in the upper right corner of the screen to save and create the new Elasticsearch data source. Nexla will now begin ingesting data from the configured endpoint and will organize any data that it finds into one or more Nexsets.

Use as a destination

Click the + icon on the Nexset that will be sent to the Elasticsearch destination, and select the Send to Destination option from the menu. Select the Elasticsearch connector from the list of available destination connectors, then select the credential that will be used to connect to the Elasticsearch organization, and click Next; or, create a new Elasticsearch credential for use in this flow.

Endpoint templates

Nexla provides pre-built templates that can be used to rapidly configure destinations to send data to common Elasticsearch endpoints. Select the endpoint to which data will be sent from the Endpoint pulldown menu. Then, click on the template in the list below to expand it, and follow the instructions to configure additional endpoint settings.

Bulk Index Documents

This endpoint performs bulk document operations (index, update, delete) on an Elasticsearch index. Use this endpoint when you need to efficiently index multiple documents, update existing documents, or delete documents in bulk.

• This endpoint sends data to Elasticsearch's bulk API. The request body should contain bulk operation commands in NDJSON (newline-delimited JSON) format following Elasticsearch's bulk API structure.
• Each record in your Nexset will be processed as a bulk operation. The bulk format requires action metadata lines followed by optional source data lines.
• The endpoint supports multiple operation types: index (create or update), create (create only), update (partial update), and delete (remove document).
• The endpoint automatically creates a corresponding data source for each bulk operation, allowing you to track indexing results.
• Bulk operations are processed efficiently, allowing you to index large volumes of documents in a single API call.

The bulk API uses NDJSON format where each line is a separate JSON object. Action lines specify the operation type and document metadata, while source lines (for index/create/update) contain the document data. For complete information about bulk operations, see the Elasticsearch Bulk API Documentation.

Index Single Document

This endpoint indexes a single document into an Elasticsearch index. Use this endpoint when you need to index individual documents, create new documents, or update specific documents.

• Optionally, enter the Document ID in the Document ID field. If provided, Elasticsearch will use this ID for the document. If left empty, Elasticsearch will auto-generate an ID.
• This endpoint sends data to Elasticsearch's index API. The request body should contain the document data in JSON format.
• Each record in your Nexset will be indexed as a separate document in the Elasticsearch index.
• The endpoint uses the PUT method to /your-index/_doc/{document_id} for indexing with a specific ID, or POST to /your-index/_doc for auto-generated IDs.

When indexing documents, Elasticsearch will create the index if it doesn't exist (if auto-create is enabled). Document IDs are unique identifiers within an index. For complete information about indexing documents, see the Elasticsearch Index API Documentation.

Update Document

This endpoint updates an existing document in an Elasticsearch index. Use this endpoint when you need to update specific fields in documents, modify document data, or perform partial updates.

• Enter the Document ID in the Document ID field. This should be the exact ID of the document you want to update in the Elasticsearch index.
• This endpoint sends data to Elasticsearch's update API. The request body should contain the update data in JSON format following Elasticsearch's update structure (using doc field for partial updates).
• Each record in your Nexset will update the specified document with the provided data.
• The endpoint uses the POST method to /your-index/_update/{document_id} to perform the update operation.

The update API supports partial updates, allowing you to modify specific fields without replacing the entire document. The request body should use the doc field to specify the fields to update. For complete information about updating documents, see the Elasticsearch Update API Documentation.

Delete Document

This endpoint deletes a document from an Elasticsearch index. Use this endpoint when you need to remove documents, clean up data, or sync deletions from other systems to Elasticsearch.

• Enter the Document ID in the Document ID field. This should be the exact ID of the document you want to delete from the Elasticsearch index.
• This endpoint sends a DELETE request to remove the specified document from the Elasticsearch index.
• Each record in your Nexset will specify a document to delete, and the endpoint will remove the specified documents from the index.
• The endpoint uses the DELETE method to /your-index/_doc/{document_id} to perform the deletion.

Document IDs are unique identifiers within an index. You can find document IDs from search results or when indexing documents. Once deleted, documents cannot be recovered unless you have backups. For complete information about deleting documents, see the Elasticsearch Delete API Documentation.

Manual configuration

Elasticsearch destinations can also be manually configured to send data to any valid Elasticsearch API endpoint. Using manual configuration, you can also configure Nexla to automatically send the response received from the Elasticsearch API after each call to a new Nexla webhook data source. Select the Advanced tab at the top of the configuration screen, and follow the instructions in Connect to Any API to configure the API method, data format, endpoint URL, request headers, attribute exclusions, record batching, and response webhooks.

Elasticsearch API URLs typically follow the format https://your-cluster.es.amazonaws.com/your-index/_bulk for bulk operations, https://your-cluster.es.amazonaws.com/your-index/_doc/{document_id} for single document operations, or https://your-cluster.es.amazonaws.com/your-index/_doc for auto-generated IDs. Use JSON format for single document operations and NDJSON (newline-delimited JSON) for bulk operations, setting the Content-Type header to application/json or application/x-ndjson accordingly. Record batching is particularly useful for the bulk endpoint, which is designed to efficiently process multiple operations in a single request.

Save & activate

Once all endpoint settings have been configured, click the Done button in the upper right corner of the screen to save and create the destination. To send the data to the configured Elasticsearch endpoint, open the destination resource menu, and select Activate.

The Nexset data will not be sent to the Elasticsearch endpoint until the destination is activated. Destinations can be activated immediately or at a later time, providing full control over data movement.