Elasticsearch Destination

Nexla's bi-directional connectors allow data to flow both to and from any location, making it simple to create a FlexFlow, DB-CDC, Spark ETL, or Replication data flow that sends data to an Elasticsearch location.

Elasticsearch

Create an Elasticsearch Destination

1. Click the + icon on the Nexset that will be sent to the Elasticsearch destination, and select the Send to Destination option from the menu.

2. Select the Elasticsearch connector from the list of available destination connectors. Then, select the credential that will be used to connect to the Elasticsearch organization, and click Next; or, create a new Elasticsearch credential for use in this flow.

3. In Nexla, Elasticsearch destinations can be created using pre-built endpoint templates, which expedite destination setup for common Elasticsearch endpoints. Each template is designed specifically for the corresponding Elasticsearch endpoint, making destination configuration easy and efficient.
   • To configure this destination using a template, follow the instructions in Configure Using a Template.

   Elasticsearch destinations can also be configured manually, allowing you to send data to Elasticsearch endpoints not included in the pre-built templates or apply further customizations to exactly suit your needs.
   • To configure this destination manually, follow the instructions in Configure Manually.

Configure Using a Template

Nexla provides pre-built templates that can be used to rapidly configure destinations to send data to common Elasticsearch endpoints. Each template is designed specifically for the corresponding Elasticsearch endpoint, making destination setup easy and efficient.

• To configure this destination using a template, select the endpoint to which data will be sent from the Endpoint pulldown menu. Then, click on the template in the list below to expand it, and follow the instructions to configure additional endpoint settings.

  Bulk Index Documents

  This endpoint performs bulk document operations (index, update, delete) on an Elasticsearch index. Use this endpoint when you need to efficiently index multiple documents, update existing documents, or delete documents in bulk.

  • This endpoint sends data to Elasticsearch's bulk API. The request body should contain bulk operation commands in NDJSON (newline-delimited JSON) format following Elasticsearch's bulk API structure.
  • Each record in your Nexset will be processed as a bulk operation. The bulk format requires action metadata lines followed by optional source data lines.
  • The endpoint supports multiple operation types: index (create or update), create (create only), update (partial update), and delete (remove document).
  • The endpoint automatically creates a corresponding data source for each bulk operation, allowing you to track indexing results.
  • Bulk operations are processed efficiently, allowing you to index large volumes of documents in a single API call.

  The bulk API uses NDJSON format where each line is a separate JSON object. Action lines specify the operation type and document metadata, while source lines (for index/create/update) contain the document data. For complete information about bulk operations, see the Elasticsearch Bulk API Documentation.

  Index Single Document

  This endpoint indexes a single document into an Elasticsearch index. Use this endpoint when you need to index individual documents, create new documents, or update specific documents.

  • Optionally, enter the Document ID in the Document ID field. If provided, Elasticsearch will use this ID for the document. If left empty, Elasticsearch will auto-generate an ID.
  • This endpoint sends data to Elasticsearch's index API. The request body should contain the document data in JSON format.
  • Each record in your Nexset will be indexed as a separate document in the Elasticsearch index.
  • The endpoint uses the PUT method to /your-index/_doc/{document_id} for indexing with a specific ID, or POST to /your-index/_doc for auto-generated IDs.

  When indexing documents, Elasticsearch will create the index if it doesn't exist (if auto-create is enabled). Document IDs are unique identifiers within an index. For complete information about indexing documents, see the Elasticsearch Index API Documentation.

  Update Document

  This endpoint updates an existing document in an Elasticsearch index. Use this endpoint when you need to update specific fields in documents, modify document data, or perform partial updates.

  • Enter the Document ID in the Document ID field. This should be the exact ID of the document you want to update in the Elasticsearch index.
  • This endpoint sends data to Elasticsearch's update API. The request body should contain the update data in JSON format following Elasticsearch's update structure (using doc field for partial updates).
  • Each record in your Nexset will update the specified document with the provided data.
  • The endpoint uses the POST method to /your-index/_update/{document_id} to perform the update operation.

  The update API supports partial updates, allowing you to modify specific fields without replacing the entire document. The request body should use the doc field to specify the fields to update. For complete information about updating documents, see the Elasticsearch Update API Documentation.

  Delete Document

  This endpoint deletes a document from an Elasticsearch index. Use this endpoint when you need to remove documents, clean up data, or sync deletions from other systems to Elasticsearch.

  • Enter the Document ID in the Document ID field. This should be the exact ID of the document you want to delete from the Elasticsearch index.
  • This endpoint sends a DELETE request to remove the specified document from the Elasticsearch index.
  • Each record in your Nexset will specify a document to delete, and the endpoint will remove the specified documents from the index.
  • The endpoint uses the DELETE method to /your-index/_doc/{document_id} to perform the deletion.

  Document IDs are unique identifiers within an index. You can find document IDs from search results or when indexing documents. Once deleted, documents cannot be recovered unless you have backups. For complete information about deleting documents, see the Elasticsearch Delete API Documentation.

Configure Manually

Elasticsearch destinations can be manually configured to send data to any valid Elasticsearch API endpoint.

Using manual configuration, you can also configure Nexla to automatically send the response received from the Elasticsearch API after each call to a new Nexla webhook data source.

API Method

1. To manually configure this destination, select the Advanced tab at the top of the configuration screen.

2. Select the API method that will be used for calls to the Elasticsearch API from the Method pulldown menu. The most common methods for Elasticsearch destinations are:

   • POST: For bulk operations, indexing documents, or updating documents (most common)
   • PUT: For indexing documents with specific IDs
   • DELETE: For removing documents

Data Format

3. Select the format in which the Nexset data will be sent to the Elasticsearch API from the Content Format pulldown menu. Nexla will automatically convert the data to the selected format for each API call.

The Elasticsearch API accepts JSON for single document operations and NDJSON (newline-delimited JSON) for bulk operations. Ensure you select the appropriate format for your Elasticsearch destination. For bulk operations, use NDJSON format. For complete information about Elasticsearch API request formats, see the Elasticsearch API Documentation.

API Endpoint URL

4. Enter the URL of the Elasticsearch API endpoint to which you want to send the Nexset data in the URL field. For update/delete operations, include the ID of the document to be updated or deleted at the end of the URL.

Elasticsearch API URLs typically follow the format: https://your-cluster.es.amazonaws.com/your-index/_bulk for bulk operations, https://your-cluster.es.amazonaws.com/your-index/_doc/{document_id} for single document operations, or https://your-cluster.es.amazonaws.com/your-index/_doc for auto-generated IDs. Replace your-cluster.es.amazonaws.com with your actual Elasticsearch cluster URL and your-index with your index name. For complete information about Elasticsearch API endpoints, see the Elasticsearch API Documentation.

Request Headers

Optional

• If Nexla should include any additional request headers in API calls to this destination, enter the headers & corresponding values as comma-separated pairs in the Request Headers field (e.g., header1:value1,header2:value2).

  You do not need to include authentication headers (basic authentication or API key headers) as these are automatically included from your credentials. However, you may need to include additional headers for specific Elasticsearch API features. The Content-Type header should be set to application/json for single document operations or application/x-ndjson for bulk operations.

Exclude Attributes from the Call

Optional

• If any record attributes in the Nexset should be omitted when sending data to this Elasticsearch destination, select the attributes from the Exclude Attributes pulldown menu.

• Any number of attributes can be selected for exclusion, and all excluded attributes will be shown in the field. To remove an attribute from the list, click the X icon next to the attribute name.

Record Batching

Optional

1. If records should be sent to this destination in batched API calls, check the box next to Would you like to batch your records together? to enable record batching.

2. Enter the maximum number of records that should be batched together in a single API call in the Batch Size field. By default, this value is set to 100.

3. Select the algorithm that will be used to group records into batches from the Grouping Algorithm pulldown menu. The sample request shown in the panel on the right will be updated to reflect the current batching settings.

Record batching is particularly useful for Elasticsearch's bulk API, which is designed to efficiently process multiple operations in a single request. Batching can significantly improve performance when indexing large volumes of documents. For complete information about Elasticsearch bulk operations, see the Elasticsearch Bulk API Documentation.