Skip to main content

Writing Data

Data destinations (also called data sinks) are the foundation of data output in the Nexla platform. They define where processed data should be written, how it should be formatted, and the configuration needed to deliver data efficiently to external systems, databases, or services.

Core Concepts​

Data destinations in Nexla provide a unified interface for writing data to various systems and formats. They handle the complexity of authentication, connection management, and data delivery, allowing you to focus on data processing rather than infrastructure concerns.

Destination Types​

Nexla supports a wide range of destination types:

  • File Systems: Amazon S3, Box, Dropbox, FTP, Google Drive, Azure Blob Storage
  • Databases: PostgreSQL, MySQL, SQL Server, Oracle, Redshift, Snowflake, BigQuery
  • Data Warehouses: Databricks, Azure Synapse, Delta Lake
  • Streaming: Kafka, Google Pub/Sub, Confluent Cloud
  • APIs: REST APIs, webhooks, custom endpoints
  • NoSQL: MongoDB, DynamoDB, Vector databases

Key Components​

Every data destination consists of:

  • Configuration: Sink type, data format, and output settings
  • Credentials: Authentication information for the target system
  • Mapping: Rules for transforming data from Nexsets to destination format
  • Scheduling: When and how often data should be written
  • Monitoring: Status tracking and performance metrics

Integration with Data Flows​

Data destinations are integral components of Nexla's data flow architecture. They receive processed data from Nexsets and deliver it to external systems according to your specifications. This integration ensures data consistency and provides end-to-end visibility into your data pipeline.

Best Practices​

To maximize the effectiveness of your data destinations:

  1. Choose the Right Type: Select destinations that match your data format and delivery requirements
  2. Optimize Mapping: Use manual mapping for complex transformations and auto-mapping for simple cases
  3. Monitor Performance: Track delivery success rates and optimize scheduling as needed
  4. Secure Credentials: Use dedicated credentials for each destination and rotate them regularly
  5. Test Connections: Validate destination connectivity before activating production flows

API Endpoints​

The Nexla API provides comprehensive endpoints for managing data destinations:

  • GET /data_sinks: List all destinations
  • POST /data_sinks: Create new destinations
  • GET /data_sinks/{id}: Retrieve destination details
  • PUT /data_sinks/{id}: Update destination configuration
  • DELETE /data_sinks/{id}: Remove destinations
  • PUT /data_sinks/{id}/activate: Activate destinations
  • PUT /data_sinks/{id}/pause: Pause destinations

Next Steps​

To get started with data destinations, explore the following sections: