Skip to main content

Monitoring

Monitoring in Nexla provides comprehensive visibility into the performance, health, and operational status of all platform resources. The system offers detailed metrics, real-time status updates, and comprehensive audit logs to help you maintain optimal data pipeline performance.

Monitoring Overview​

Nexla's monitoring capabilities enable you to track resource performance, identify issues proactively, and maintain data quality across your entire infrastructure. These monitoring methods provide full insight into how resources are functioning, their operational history, and complete audit trails.

Core Monitoring Features​

The monitoring system provides several key capabilities for maintaining platform health and performance.

Resource Status Monitoring​

Track the operational status of all resources in real-time:

  • Active Resources: Monitor currently running data sources, destinations, and flows
  • Status Changes: Receive notifications when resource status changes
  • Health Checks: Automated health monitoring for all platform components
  • Performance Metrics: Track resource performance and efficiency

Historical Data Analysis​

Analyze resource performance over time to identify trends and patterns:

  • Performance Trends: Track resource performance over days, weeks, and months
  • Usage Patterns: Understand how resources are being utilized
  • Capacity Planning: Plan resource allocation based on historical usage
  • Optimization Opportunities: Identify areas for performance improvement

Audit and Compliance​

Maintain comprehensive audit trails for compliance and security:

  • Access Logs: Track all resource access and modifications
  • Change History: Monitor configuration changes and updates
  • User Activity: Track user actions across the platform
  • Compliance Reporting: Generate reports for regulatory requirements

Notifications​

Notification resources provide real-time alerts about events and status changes related to your data resources. The system automatically generates notifications for important events, allowing you to respond quickly to issues and opportunities.

Notification System Overview​

The notification system keeps you informed about critical events across your data infrastructure, enabling proactive management and rapid response to issues.

List All Notifications​

Retrieve all notifications to get a comprehensive view of system events and status changes.

List All Notifications Endpoint​

To retrieve all your notifications:

GET /notifications

Filtering Options​

Use query parameters to filter notifications based on your needs:

List All Notifications: Request
GET /notifications?read=0

List All Notifications Response​

The response includes comprehensive notification information with filtering options:

List All Notifications: Response
[
  {
    "id": 5007,
    "owner": {
      "id": 42,
      "full_name": "John Smith",
      "email": "john.smith@example.com"
    },
    "org": {
      "id": 101,
      "name": "Acme Corporation",
      "email_domain": "acme.com"
    },
    "access_roles": ["owner"],
    "level": "INFO",
    "resource_id": 1002,
    "resource_type": "SOURCE",
    "message_id": 0,
    "message": "Data source connected successfully",
    "read_at": null,
    "updated_at": "2023-01-15T21:04:48.000Z",
    "created_at": "2023-01-15T21:04:48.000Z"
  },
  {
    "id": 5008,
    "owner": {
      "id": 42,
      "full_name": "John Smith",
      "email": "john.smith@example.com"
    },
    "org": {
      "id": 101,
      "name": "Acme Corporation",
      "email_domain": "acme.com"
    },
    "access_roles": ["owner"],
    "level": "INFO",
    "resource_id": 3001,
    "resource_type": "SET",
    "message_id": 0,
    "message": "New datasets available for processing",
    "read_at": null,
    "updated_at": "2023-01-15T21:05:48.000Z",
    "created_at": "2023-01-15T21:05:48.000Z"
  }
]

Filtered Notifications​

Use advanced filtering to focus on specific types of notifications and events.

Filtering by Resource Type and Level​

Filter notifications by resource type, severity level, and specific resource ID:

Filtered Notifications: Request
GET /notifications?read=0&resource_type=SOURCE&resource_id=1002&level=ERROR

Filtering Parameters​

Available filtering options for notifications:

  • read: Filter by read status (0 for unread, 1 for read)
  • resource_type: Filter by resource type (SOURCE, SET, SINK, FLOW)
  • level: Filter by notification level (DEBUG, INFO, WARN, ERROR)
  • resource_id: Filter by specific resource ID
  • from: Filter by start date/time
  • to: Filter by end date/time

Notification Levels​

Understanding notification levels helps you prioritize responses and focus on critical issues.

Debug Level​

Debug notifications provide detailed technical information:

  • Development Support: Helpful during development and testing
  • Troubleshooting: Detailed information for problem resolution
  • Performance Analysis: Technical details for optimization
  • Low Priority: Generally not requiring immediate attention

Info Level​

Info notifications provide general operational information:

  • Status Updates: Normal operational status changes
  • Resource Events: Successful operations and completions
  • System Updates: Platform and configuration changes
  • Medium Priority: Monitor for operational awareness

Warning Level​

Warning notifications indicate potential issues that may require attention:

  • Performance Issues: Slower than expected performance
  • Resource Constraints: Approaching capacity limits
  • Configuration Warnings: Non-critical configuration issues
  • High Priority: Monitor closely and address proactively

Error Level​

Error notifications indicate problems that require immediate attention:

  • Connection Failures: Failed connections to external systems
  • Processing Errors: Data processing failures and exceptions
  • Resource Failures: Complete resource failures and outages
  • Critical Priority: Immediate attention and resolution required

Mark Notifications as Read​

Update notification status to track which alerts have been reviewed and addressed.

Mark as Read Endpoint​

To mark notifications as read:

PUT /notifications/{notification_id}/read
Mark Notification as Read: Request
PUT /notifications/5007/read

Mark as Unread Endpoint​

To mark notifications as unread for later review:

PUT /notifications/{notification_id}/unread
Mark Notification as Unread: Request
PUT /notifications/5007/unread

Bulk Notification Operations​

Perform operations on multiple notifications simultaneously for efficient management.

Bulk Mark as Read​

Mark multiple notifications as read in a single operation:

PUT /notifications/bulk_read
Bulk Mark as Read: Request
PUT /notifications/bulk_read

{
  "notification_ids": [5007, 5008, 5009],
  "read": true
}

Bulk Mark as Unread​

Mark multiple notifications as unread for later review:

Bulk Mark as Unread: Request
PUT /notifications/bulk_read

{
  "notification_ids": [5007, 5008, 5009],
  "read": false
}

Resource-Specific Monitoring​

Different resource types have specialized monitoring capabilities tailored to their specific functions and requirements.

Data Source Monitoring​

Monitor data ingestion performance and health:

  • Ingestion Metrics: Track data volume, file counts, and processing rates
  • Connection Status: Monitor connectivity to external data sources
  • Schema Detection: Track automatic schema detection and validation
  • Error Handling: Monitor ingestion errors and retry attempts

Data Destination Monitoring​

Monitor data output performance and delivery:

  • Delivery Metrics: Track successful data deliveries and failures
  • Performance Monitoring: Monitor output processing speed and efficiency
  • Error Tracking: Track delivery failures and retry attempts
  • Capacity Planning: Monitor destination capacity and utilization

Flow Monitoring​

Monitor data processing pipeline performance:

  • Flow Status: Track active, paused, and error states
  • Processing Metrics: Monitor data processing rates and volumes
  • Resource Utilization: Track CPU, memory, and network usage
  • Error Handling: Monitor processing errors and recovery

Metrics and Performance​

Comprehensive metrics provide detailed insights into resource performance and efficiency.

Performance Metrics​

Track key performance indicators across all resources:

  • Throughput: Data processing rates and volumes
  • Latency: Response times and processing delays
  • Efficiency: Resource utilization and optimization
  • Reliability: Success rates and error frequencies

Resource Metrics​

Monitor resource-specific performance indicators:

  • Data Sources: Ingestion rates, file processing, schema detection
  • Data Destinations: Delivery rates, output processing, error handling
  • Data Flows: Processing speed, resource usage, pipeline efficiency
  • Credentials: Connection success rates, authentication performance

Historical Analysis​

Analyze performance trends over time to identify patterns and opportunities:

  • Trend Analysis: Long-term performance patterns and changes
  • Seasonal Variations: Identify recurring patterns and cycles
  • Capacity Planning: Plan resource allocation based on usage trends
  • Optimization: Identify areas for performance improvement

Audit and Compliance​

Comprehensive audit trails ensure compliance and security across your platform.

Audit Logs​

Track all system activities and changes:

  • Access Logs: Monitor user access and resource usage
  • Configuration Changes: Track all resource modifications
  • Security Events: Monitor authentication and authorization
  • Compliance Reporting: Generate reports for regulatory requirements

Change Tracking​

Monitor all modifications to resources and configurations:

  • Configuration Changes: Track settings and parameter updates
  • Access Modifications: Monitor permission changes and grants
  • Resource Lifecycle: Track creation, modification, and deletion
  • User Activity: Monitor all user actions and operations

Compliance Features​

Built-in compliance capabilities for regulatory requirements:

  • Data Governance: Track data lineage and transformations
  • Access Control: Monitor user permissions and access patterns
  • Audit Trails: Complete history of all system activities
  • Reporting: Generate compliance reports and documentation

Monitoring Integration​

Integrate monitoring data with external systems for comprehensive observability.

External Monitoring Systems​

Connect Nexla monitoring to your existing monitoring infrastructure:

  • Metrics Export: Export metrics to external monitoring systems
  • Alert Integration: Forward notifications to external alerting systems
  • Dashboard Integration: Display Nexla metrics in external dashboards
  • Log Aggregation: Forward logs to centralized logging systems

API Integration​

Use monitoring APIs to build custom monitoring solutions:

  • Custom Dashboards: Build specialized monitoring interfaces
  • Automated Responses: Trigger automated actions based on metrics
  • Custom Alerts: Implement specialized alerting logic
  • Integration Workflows: Connect monitoring to business processes

Best Practices​

To effectively implement monitoring in your Nexla platform:

  1. Define Key Metrics: Identify the most important performance indicators
  2. Set Appropriate Thresholds: Configure alerts for meaningful events
  3. Regular Review: Periodically review monitoring data and trends
  4. Automated Response: Implement automated responses for common issues
  5. Documentation: Maintain clear documentation of monitoring procedures

Monitoring Workflows​

Common monitoring workflows for different operational scenarios.

Daily Operations​

Routine monitoring for daily operations:

  1. Status Check: Review overall platform health and status
  2. Performance Review: Check key performance metrics
  3. Error Review: Review and address any errors or warnings
  4. Capacity Planning: Monitor resource utilization and plan capacity

Incident Response​

Monitoring during incident response:

  1. Alert Assessment: Evaluate and prioritize alerts
  2. Impact Analysis: Assess the scope and impact of issues
  3. Resolution Tracking: Monitor progress toward resolution
  4. Post-Incident Review: Analyze monitoring data for lessons learned

Performance Optimization​

Monitoring for performance improvement:

  1. Baseline Establishment: Establish performance baselines
  2. Trend Analysis: Identify performance trends and patterns
  3. Optimization Planning: Plan performance improvements
  4. Impact Measurement: Measure the impact of optimizations

Error Handling​

Common monitoring issues and solutions:

  • Missing Metrics: Verify resource configuration and monitoring setup
  • Alert Fatigue: Adjust thresholds and filtering to reduce noise
  • Performance Impact: Monitor monitoring system performance
  • Data Accuracy: Validate monitoring data accuracy and completeness

After setting up monitoring, you may need to:

Configure Alerts​

PUT /notifications/config
GET /notifications/config

Export Metrics​

GET /metrics/export
GET /metrics/{resource_type}/{resource_id}

View Audit Logs​

GET /audit_logs
GET /audit_logs/{resource_type}/{resource_id}