Skip to main content

Destination Monitoring

Destination monitoring provides comprehensive visibility into the performance, health, and operational status of your data output processes. These monitoring capabilities help you track data delivery success, identify issues, and optimize output performance.

Lifetime Write Metrics​

Lifetime metrics provide a comprehensive view of all data written through a destination since its creation. This gives you a complete picture of the destination's historical performance and data volume.

Lifetime Metrics Endpoint​

Lifetime Write Metrics: Request
GET /data_sinks/{destination_id}/metrics

Response Structure​

Lifetime Write Metrics: Response
{
  "status": 200,
  "metrics": {
    "records": 1250000,
    "size": 2048576000,
    "files_written": 1500,
    "errors": 25,
    "last_write": "2023-01-15T10:30:00.000Z"
  }
}

Lifetime Metrics Benefits​

These metrics help you understand:

  • Total Data Volume: Complete picture of data delivered
  • Historical Performance: Long-term trends and patterns
  • Error Rates: Overall reliability and quality
  • Resource Utilization: Data processing efficiency

Aggregated Write Metrics​

Aggregated metrics provide time-based views of output performance, allowing you to analyze trends, identify patterns, and monitor daily, weekly, or monthly performance.

Daily Aggregation Endpoint​

Daily Write Metrics: Request
GET /data_sinks/{destination_id}/metrics?aggregate=1

Optional Query Parameters:
?from=2023-01-01T00:00:00&to=2023-01-31T23:59:59&page=1&size=100

Response Structure​

Daily Write Metrics: Response
{
  "status": 200,
  "metrics": [
    {
      "time": "2023-01-15",
      "records": 53054,
      "size": 12476341,
      "files_written": 45,
      "errors": 2
    },
    {
      "time": "2023-01-16",
      "records": 66618,
      "size": 15829589,
      "files_written": 52,
      "errors": 0
    },
    {
      "time": "2023-01-17",
      "records": 25832,
      "size": 6645994,
      "files_written": 18,
      "errors": 1
    }
  ],
  "pagination": {
    "page": 1,
    "size": 100,
    "total": 31
  }
}

Aggregation Granularity​

You can adjust the time granularity of metrics:

  • Daily: ?aggregate=1 (default)
  • Hourly: ?aggregate=hour
  • Weekly: ?aggregate=week
  • Monthly: ?aggregate=month

Write Run Metrics​

Write run metrics provide detailed information about individual output cycles, giving you visibility into the performance of each data delivery event.

Run Summary Endpoint​

Write Run Metrics: Request
GET /data_sinks/{destination_id}/metrics/run_summary

Optional Query Parameters:
?from=2023-01-15T00:00:00&to=2023-01-15T23:59:59&page=1&size=50

Response Structure​

Write Run Metrics: Response
{
  "status": 200,
  "metrics": {
    "1673776800000": {
      "run_id": "1673776800000",
      "start_time": "2023-01-15T06:00:00.000Z",
      "end_time": "2023-01-15T06:15:00.000Z",
      "records": 1364,
      "size": 971330,
      "files_written": 12,
      "errors": 0,
      "status": "completed"
    },
    "1673780400000": {
      "run_id": "1673780400000",
      "start_time": "2023-01-15T07:00:00.000Z",
      "end_time": "2023-01-15T07:08:00.000Z",
      "records": 330,
      "size": 235029,
      "files_written": 3,
      "errors": 0,
      "status": "completed"
    }
  }
}

Run Metrics Analysis​

Use run metrics to analyze:

  • Performance Patterns: Identify slow or fast output cycles
  • Error Tracking: Monitor error rates per output run
  • Resource Usage: Track processing time and efficiency
  • Scheduling Optimization: Optimize output frequency

File Processing Metrics​

For file-based destinations, you can monitor detailed statistics about file processing, including success rates, failure reasons, and processing status.

File Statistics Endpoint​

File Processing Metrics: Request
GET /data_sinks/{destination_id}/metrics/files_stats

Optional Query Parameters:
?from=2023-01-15T00:00:00&to=2023-01-15T23:59:59&status=all

Response Structure​

File Processing Metrics: Response
{
  "status": 200,
  "file_stats": {
    "total_files": 150,
    "processed": 142,
    "failed": 5,
    "queued": 3,
    "processing": 0,
    "status_breakdown": {
      "success": 142,
      "validation_error": 3,
      "format_error": 2,
      "access_denied": 0
    }
  },
  "recent_files": [
    {
      "file_path": "daily/customer_data_2023-01-15.csv",
      "status": "processed",
      "records": 1250,
      "size": 256000,
      "processing_time": 45,
      "timestamp": "2023-01-15T10:30:00.000Z"
    }
  ]
}

Real-time Monitoring​

Monitor destinations in real-time to get immediate alerts and status updates.

Health Check Endpoint​

Destination Health Check: Request
GET /data_sinks/{destination_id}/health

Health Status Response​

Destination Health Status: Response
{
  "status": "healthy",
  "last_check": "2023-01-15T10:30:00.000Z",
  "connection_status": "connected",
  "credential_status": "valid",
  "last_successful_write": "2023-01-15T10:00:00.000Z",
  "write_lag": 1800,
  "alerts": []
}

Performance Analytics​

Analyze output performance to optimize your data delivery processes.

Performance Metrics​

Track key performance indicators:

  • Output Rate: Records processed per second
  • Throughput: Data volume processed per time unit
  • Latency: Time from data availability to delivery
  • Efficiency: Resource utilization and cost per record

Performance Optimization​

Use metrics to:

  • Adjust Scheduling: Optimize output frequency
  • Scale Resources: Add or remove processing capacity
  • Improve Configuration: Fine-tune destination settings
  • Monitor Costs: Track resource consumption

Alerting and Notifications​

Set up automated alerts for monitoring critical issues:

Alert Types​

  • Output Failures: Failed data delivery attempts
  • Performance Degradation: Slower than expected processing
  • Data Quality Issues: High error rates or validation failures
  • Resource Constraints: Memory, CPU, or storage limitations

Alert Configuration​

Configure alerts through:

  • Thresholds: Set performance and error rate limits
  • Channels: Email, webhook, or integration notifications
  • Escalation: Automatic escalation for critical issues
  • Suppression: Temporarily disable alerts during maintenance

Data Quality Monitoring​

Monitor the quality and consistency of delivered data:

Quality Metrics​

  • Completeness: Percentage of non-null values
  • Accuracy: Data validation against business rules
  • Consistency: Format and value uniformity
  • Timeliness: Data freshness and update frequency

Quality Alerts​

Set up alerts for:

  • Data Drift: Unexpected changes in data structure
  • Anomalies: Unusual patterns or values
  • Missing Data: Gaps in expected data delivery
  • Validation Failures: Data that doesn't meet quality standards

Best Practices​

To maximize the value of destination monitoring:

  1. Set Baselines: Establish normal performance ranges
  2. Monitor Trends: Track performance over time
  3. Set Alerts: Configure automated notifications
  4. Analyze Patterns: Identify recurring issues
  5. Optimize Continuously: Use insights to improve performance
  6. Document Issues: Keep records of problems and solutions

Error Handling​

Common monitoring issues and solutions:

  • Missing Metrics: Verify destination is active and delivering data
  • Inconsistent Data: Check for configuration changes or destination issues
  • Performance Degradation: Investigate resource constraints or bottlenecks
  • Connection Failures: Verify credentials and network connectivity
  • Data Quality Issues: Review validation rules and data sources

After monitoring your destinations, you may need to:

Update Configuration​

PUT /data_sinks/{destination_id}

Test Connection​

PUT /data_sinks/{destination_id}/probe/authenticate

Validate Configuration​

POST /data_sinks/{destination_id}/config/validate

Activate or Pause​

PUT /data_sinks/{destination_id}/activate
PUT /data_sinks/{destination_id}/pause