Skip to main content

Monitoring

Nexla maintains and provides detailed monitoring information for all Nexla resources. Using these monitoring methods you can get full insight into how resources are functioning, their history, and full audit log.

Additionally, these monitoring endpoints can in turn be made Nexla sources if you wish to automate data flows based on monitoring events.

The items in this section are common to all Nexla resources. In addition to these endpoints, Nexla Sources and Destinations have some special monitoring capabilities, for ex: file sources have methods to track ingestion history of each file. Refer to the relevant monitoring sections in the Reading Data and Writing Data pages of this site.

Notifications​

Notification resources describe messages to be relayed to users, usually regarding events or status changes related to a particular data resource. Notification creation is currently limited to the Nexla platform itself. API users can read notifications and mark them read or unread only.

List All Notifications​

Issue a GET request to the /notifications endpoint to retrieve a list of all your notifications. Set the read query parameter to 0 to fetch only notifications that have not been read yet, or to 1 to fetch only those that have been read.

List All Notifications: Request
GET /notifications?read=0
List All Notifications: Response
[
  {
    "id": 5007,
    "owner": {
      "id": 2,
      "full_name": "Jeff Williams",
      "email": "jcw@nexla.com"
    },
    "org": {
      "id": 1,
      "name": "Nexla",
      "email_domain": "nexla.com"
    },
    "access_roles": ["owner"],
    "level": "INFO",
    "resource_id": 5002,
    "resource_type": "SOURCE",
    "message_id": 0,
    "message": "Data source connected",
    "read_at": null,
    "updated_at": "2017-02-27T21:04:48.000Z",
    "created_at": "2017-02-27T21:04:48.000Z"
  },
  {
    "id": 5008,
    "owner": {
      "id": 2,
      "full_name": "Jeff Williams",
      "email": "jcw@nexla.com"
    },
    "org": {
      "id": 1,
      "name": "Nexla",
      "email_domain": "nexla.com"
    },
    "access_roles": ["owner"],
    "level": "INFO",
    "resource_id": 5100,
    "resource_type": "SET",
    "message_id": 0,
    "message": "New datasets available",
    "read_at": null,
    "updated_at": "2017-02-27T21:05:48.000Z",
    "created_at": "2017-02-27T21:05:48.000Z"
  }
]

List Notifications by resource_type, level and resource_id​

Issue a GET request to the /notifications endpoint to retrieve a list of all your notifications by optional filters resource_type('SOURCE','SET','SINK'), level('DEBUG','INFO','WARN','ERROR') and resource_id.

List Notification With Filters: Request
GET /notifications?read=0&resource_type={resource_type}&resource_id={resource_id}&level=ERROR
List Notification With Filters: Response
[
  {
    "id": 5007,
    "owner": {
      "id": 2,
      "full_name": "Jeff Williams",
      "email": "jcw@nexla.com"
    },
    "org": {
      "id": 1,
      "name": "Nexla",
      "email_domain": "nexla.com"
    },
    "access_roles": ["owner"],
    "level": "ERROR",
    "resource_id": 6889,
    "resource_type": "SOURCE",
    "message_id": 0,
    "message": "Data source connected",
    "read_at": null,
    "updated_at": "2017-02-27T21:04:48.000Z",
    "created_at": "2017-02-27T21:04:48.000Z"
  },
  {
    "id": 5008,
    "owner": {
      "id": 2,
      "full_name": "Jeff Williams",
      "email": "jcw@nexla.com"
    },
    "org": {
      "id": 1,
      "name": "Nexla",
      "email_domain": "nexla.com"
    },
    "access_roles": ["owner"],
    "level": "ERROR",
    "resource_id": 6889,
    "resource_type": "SOURCE",
    "message_id": 0,
    "message": "New subscriptions available",
    "read_at": null,
    "updated_at": "2017-02-27T21:05:48.000Z",
    "created_at": "2017-02-27T21:05:48.000Z"
  }
]

List Notifications with sort_by and sort_order​

Issue a GET request to the /notifications endpoint to retrieve a list of all your notifications sorted by columns with sorted order(DESC/ASC). You can send multiple columns by comma separated in sort_by param. Default values of sort_by is created_at and for sort_order is DESC.

List Notifications With Sorting: Request
GET /notifications?read=0&resource_type={resource_type}&resource_id={resource_id}&level=ERROR&sort_by=created_at,id&sort_order=DESC
List Notifications With Sorting: Response
[
  {
    "id": 5007,
    "owner": {
      "id": 2,
      "full_name": "Jeff Williams",
      "email": "jcw@nexla.com"
    },
    "org": {
      "id": 1,
      "name": "Nexla",
      "email_domain": "nexla.com"
    },
    "access_roles": ["owner"],
    "level": "ERROR",
    "resource_id": 6889,
    "resource_type": "SOURCE",
    "message_id": 0,
    "message": "Data source connected",
    "read_at": null,
    "updated_at": "2017-02-27T21:04:48.000Z",
    "created_at": "2017-02-27T21:04:48.000Z"
  },
  {
    "id": 5008,
    "owner": {
      "id": 2,
      "full_name": "Jeff Williams",
      "email": "jcw@nexla.com"
    },
    "org": {
      "id": 1,
      "name": "Nexla",
      "email_domain": "nexla.com"
    },
    "access_roles": ["owner"],
    "level": "ERROR",
    "resource_id": 6889,
    "resource_type": "SOURCE",
    "message_id": 0,
    "message": "New subscriptions available",
    "read_at": null,
    "updated_at": "2017-02-27T21:05:48.000Z",
    "created_at": "2017-02-27T21:05:48.000Z"
  }
]

Notification Counts​

Issue a GET request to the /notifications/count endpoint to retrieve count of your notifications. Set the read query parameter to 0 to fetch only notifications that have not been read yet, or to 1 to fetch only those that have been read. If you don't send read query param, will fetch all(read/unread both) notifications.

Notification Counts: Request
GET /notifications/count?read=0
Notification Counts: Response
{
  "count": 1623
}

Mark Notification Read/Unread​

To mark a notification as "read", make a PUT request to the /notifications/<notification_id>/mark_read endpoint. To mark it "unread", make a PUT request to /notifications/<notification_id>/mark_unread.

Marking a notification as "read" updates its read_at attribute to the time of the /mark_read request. Marking it "unread" clears the read_at attribute.

A successful PUT call to /notifications/<notification_id>/mark_read returns the complete notification resource with the read_at attributed updated to the time of the request

Mark Notification Read/Unread: Request
PUT /notifications/{notification_id}/mark_read
Mark Notification Read/Unread: Response
{
  "id": 5008,
  "owner": {
    "id": 2,
    "full_name": "Jeff Williams",
    "email": "jcw@nexla.com"
  },
  "org": {
    "id": 1,
    "name": "Nexla",
    "email_domain": "nexla.com"
  },
  "access_roles": ["owner"],
  "level": "INFO",
  "resource_id": null,
  "resource_type": null,
  "message_id": 0,
  "message": "New subscriptions available",
  "read_at": "2017-02-27T21:19:20.562Z",
  "updated_at": "2017-02-27T21:19:20.563Z",
  "created_at": "2017-02-27T21:05:48.000Z"
}

Delete Notification​

Issue a DELETE call to the /notificatons/{notification_id} endpoint to delete a notification.

View Erroneous Records​

When Nexla is not able to move move any ingested record across a dataset or write it out to a destination, not only does it generate an error notification to inform the user, it also quarantines the error records out as quarantined samples. Doing so enables the user to setup workflows for reviewing the error records, fixing the issue and writing them out to the destination after fixes.

Users can fetch quarantine samples to review the current qurantined records. But since Nexla does not retain any user data, these methods will only return freshly quarantined samples that have yet to be flushed out of Nexla.

Each error data sample is structured as follows:

AttributeDescription
nexlaMetaDataInformation about the record (where it came from, when it was ingested, etc), handy for tracing its entry point into Nexla all the way to ingestion.
errorDetailed information about reason for error.
rawMessageThe record that failed to move across that resource. For ex, it is the raw line of data in source file for data source ingestion error, the input record to a transform for a data set transform error, or the output record that Nexla failed to write out to a data destination.
Fetch Error Samples: Request
POST  /{resource_type}/{resource_id}/probe/quarantine/sample.

...
Example: For Quarantine Samples of Source
POST /data_sources/{data_source_id}/probe/quarantine/sample
{
  "page":1,
  "per_page":10,
  "start_time":1546916322076,
  "end_time":1547081941493
}
...
Replace /data_sources/{data_source_id} with /data_sets/{data_set_id} or /data_sinks/{data_sink_id}
Fetch Error Samples: Response
{
  "status": 200,
  "message": "Ok",
  "output": {
    "data": [
      {
        "nexlaMetaData": {
          "sourceType": "S3",
          "ingestTime": 1546916322079,
          "sourceOffset": 43808,
          "sourceKey": "to_nexla-s3-bucket/airtimes_201808.csv",
          "bucket": "nexla-s3-bucket",
          "topic": "dataset-19641-source-9704",
          "resourceType": "SOURCE",
          "resourceId": 9704,
          "nexlaUUID": null,
          "trackerId": {
            "source": {
              "id": 9704,
              "source": "airtimes_201808.csv",
              "offset": 43808,
              "recordNumber": "43808",
              "version": 1,
              "initialIngestTimestamp": 1546916322079
            },
            "sets": [],
            "sink": null
          },
          "eof": false,
          "lastModified": null,
          "runId": 1546907944699,
          "tags": null
        },
        "error": {
          "message": "Too many entries: expected at most 6 (value #6 (2 chars) \"15\")\n at [Source: (BufferedReader); line: 43809, column: 128]",
          "lineNumber": 43808,
          "exceptionTrace": "com.fasterxml.jackson.databind.RuntimeJsonMappingException",
          "responseBody": null
        },
        "rawMessage": {
          "message": "2018-08-31 18:38:30.000000,BOOM,Boomerang,\"Karis (15) \"\"Deal of a Lifetime\"\" $5 off endcard  (sparkle)\",\"BE COOL, SCOOBY-DOO!\",15"
        },
        "time": 1546916322080
      },
      {
        "nexlaMetaData": {
          "sourceType": "S3",
          "ingestTime": 1546916322078,
          "sourceOffset": 43788,
          "sourceKey": "to_nexla-s3-bucket/airtimes_201808.csv",
          "bucket": "nexla-s3-bucket",
          "topic": "dataset-19641-source-9704",
          "resourceType": "SOURCE",
          "resourceId": 9704,
          "nexlaUUID": null,
          "trackerId": {
            "source": {
              "id": 9704,
              "source": "airtimes_201808.csv",
              "offset": 43788,
              "recordNumber": "43788",
              "version": 1,
              "initialIngestTimestamp": 1546916322078
            },
            "sets": [],
            "sink": null
          },
          "eof": false,
          "lastModified": null,
          "runId": 1546907944699,
          "tags": null
        },
        "error": {
          "message": "Too many entries: expected at most 6 (value #6 (2 chars) \"15\")\n at [Source: (BufferedReader); line: 43789, column: 128]",
          "lineNumber": 43788,
          "exceptionTrace": "com.fasterxml.jackson.databind.RuntimeJsonMappingException",
          "responseBody": null
        },
        "rawMessage": {
          "message": "2018-08-31 18:18:22.000000,BOOM,Boomerang,\"Karis (15) \"\"Deal of a Lifetime\"\" $5 off endcard  (sparkle)\",\"BE COOL, SCOOBY-DOO!\",15"
        },
        "time": 1546916322079
      },
      {
        "nexlaMetaData": {
          "sourceType": "S3",
          "ingestTime": 1546916322074,
          "sourceOffset": 43664,
          "sourceKey": "to_nexla-s3-bucket/airtimes_201808.csv",
          "bucket": "nexla-s3-bucket",
          "topic": "dataset-19641-source-9704",
          "resourceType": "SOURCE",
          "resourceId": 9704,
          "nexlaUUID": null,
          "trackerId": {
            "source": {
              "id": 9704,
              "source": "airtimes_201808.csv",
              "offset": 43664,
              "recordNumber": "43664",
              "version": 1,
              "initialIngestTimestamp": 1546916322074
            },
            "sets": [],
            "sink": null
          },
          "eof": false,
          "lastModified": null,
          "runId": 1546907944699,
          "tags": null
        },
        "error": {
          "message": "Too many entries: expected at most 6 (value #6 (5 chars) \"DIVE\"\")\n at [Source: (BufferedReader); line: 43665, column: 115]",
          "lineNumber": 43664,
          "exceptionTrace": "com.fasterxml.jackson.databind.RuntimeJsonMappingException",
          "responseBody": null
        },
        "rawMessage": {
          "message": "2018-08-31 15:56:44.000000,QUBO,QUBO,\"Vanessa Mashup Vacations...Weddings...Car\"\" (15) Download Now\",\"DIVE, OLLY, DIVE\",15"
        },
        "time": 1546916322076
      }
    ],
    "meta": {
      "currentPage": 1,
      "totalCount": 3,
      "pageCount": 1
    }
  }
}

Auto Export Erroneous Records​

Users can configure Nexla to automatically flush out quarantined samples to any file system of their choice. This is called as Error Data location, and the configuration settings and controls are the same as that for any file type destination.

Furthermore, error data locations can be configured uniquely for any level: resource(source, dataset, destination), user, and org. For, you can configure Nexla such that quarantine data for a critical source is sent to one file location, for all other resources of a user is sent to another location (folder hierarchy is automatically namespaced by Nexla to separate out resources), and for all resources of all other users in the org to a catch all org wide location (folder hierarchy namespaced by users and resources).

Set Error Data Location​

Issue a POST call to the <resource_type>/<resource_id>/quarantine_settings endpoint to configure that resources's error data location. data_credentials, frequency_cron, and path attributes are required.

For ex, this call configures error data location for user user_id. Replace/users/{user_id} with /data_sources/{data_source_id}, /data_sets/{data_set_id}, /data_sinks/{data_sink_id}, or /orgs/{org_id} for corresponding resource settings.

Set Error Data Location: Request
POST /users/{user_id}/quarantine_settings
...
{
  "data_credentials": 5010,
  "config": {
    "start.cron": "0 0 * 1/1 * ? *",
    "path": "test.nexla.com/nexla"
  }
}
Set Error Data Location: Response
{
    "id": 2,
    "owner": {
        "id": 27,
        "full_name": "Jeff Williams",
        "email": "jcw@nexla.com"
    },
    "org": {
        "id": 1,
        "name": "Nexla",
        "email_domain": "nexla.com",
        "email": null
    },
    "resource_type": "USER",
    "resource_id": 27,
    "config": {
        "start.cron": "0 0 * 1/1 * ? *",
        "bucket": "test.nexla.com",
        "prefix": "nexla"
    },
    "data_credentials_id": 5010,
    "credentials_type": "s3",
    "data_credentials": {
        "id": 5010,
        ...
    }
}

Update Error Data Location​

Issue a PUT call to the <resource_type>/<resource_id>/quarantine_settings endpoint to update the error data location.

For ex, this call updates error data location for user user_id. Replace /users/{user_id} with /data_sources/{data_source_id}, /data_sets/{data_set_id}, /data_sinks/{data_sink_id}, or /orgs/{org_id} for corresponding resource settings.

Update Error Data Location: Request
PUT /users/{user_id}/quarantine_settings
{
  "config": {
      "start.cron": "0 0 * 1/1 * ? *",
      "bucket": "test.nexla.com",
      "prefix": "nexla_new"
  }
}
Update Error Data Location: Response
{
    "id": 2,
    "owner": {
        "id": 27,
        "full_name": "Jeff Williams",
        "email": "jcw@nexla.com"
    },
    "org": {
        "id": 1,
        "name": "Nexla",
        "email_domain": "nexla.com",
        "email": null
    },
    "resource_type": "USER",
    "resource_id": 27,
    "config": {
        "start.cron": "0 0 * 1/1 * ? *",
        "bucket": "test.nexla.com",
        "prefix": "nexla_new"
    },
    "data_credentials_id": 5010,
    "credentials_type": "s3",
    "data_credentials": {
        "id": 5010,
       ...
    }
}

Show Current Error Data Configuration​

Issue a GET call to the <resource_type>/<resource_id>/quarantine_settings endpoint to show error data location.

For ex, this call displays error data location for user user_id. Replace /users/{user_id} with /data_sources/{data_source_id}, /data_sets/{data_set_id}, /data_sinks/{data_sink_id}, or /orgs/{org_id} for corresponding resource settings.

Show Current Error Data Configuration: Request
GET /users/{user_id}/quarantine_settings
Show Current Error Data Configuration: Response
{
    "id": 2,
    "owner": {
        "id": 27,
        "full_name": "Jeff Williams",
        "email": "jcw@nexla.com"
    },
    "org": {
        "id": 1,
        "name": "Nexla",
        "email_domain": "nexla.com",
        "email": null
    },
    "resource_type": "USER",
    "resource_id": 27,
    "config": {
        "start.cron": "0 0 * 1/1 * ? *",
        "bucket": "test.nexla.com",
        "prefix": "nexla_new"
    },
    "data_credentials_id": 5010,
    "credentials_type": "s3",
    "data_credentials": {
        "id": 5010,
        ...
    }
}

List All Error Data configurations​

Issue a GET call to the /quarantine_settings endpoint to list all error data locations for a resource. This is specially handy for reviewing all export locations of your account when you have configured custom error data export locations for different resources.

List All Error Data Configurations: Request
GET /quarantine_settings?resource_type={resource_type}&resource_id={resource_id}
...
Example: /quarantine_settings?resource_type=user&resource_id={user_id}
List All Error Data Configurations: Response
[
    {
        "id": 2,
        "owner": {
            "id": 27,
            "full_name": "Jeff Williams",
            "email": "jcw@nexla.com"
        },
        "org": {
            "id": 1,
            "name": "Nexla",
            "email_domain": "nexla.com",
            "email": null
        },
        "resource_type": "SOURCE",
        "resource_id": 5002,
        "config": {
                "start.cron": "0 0 * 1/1 * ? *",
                "bucket": "test.nexla.com",
                "prefix": "nexla_new"
        },
        "data_credentials_id": 5010,
        "credentials_type": "s3",
        "data_credentials": {
            "id": 5010,
            ...
        }
    }
]

Reviewing Exported Error Data Files​

Not only can you automatically configure Nexla to export error samples to a file system, you can also fetch a list of exported error files. This is very similar to list of files written by Nexla to any regular file destination.

Issue a GET request to the {resource_type}/{resource_id}/metrics/quarantine_files endpoint to retrieve all quarantine files for a resource. You can also send optional parameters from(start time), to(end time), data_credentials_id, quarantine_setting_id to add filter.

For ex, this call returns all error files written out for data source 5001. Note that the user need not have configured a custom quarantine location for data source 5001 for this to work, they might just have configured a catch all account wide export location.

Sample Exported Error Data File: Request
GET /data_sources/5001/metrics/quarantine_files?from=2017-09-25T02:25:26&to=2017-09-28T02:25:26
Sample Exported Error Data File: Response
{
  "status": 200,
  "metrics": {
    "data": [
      {
        "resourceId": 5002,
        "owner_id": 27,
        "recordCount": 5,
        "created_at": "2018-08-23T03:02:34.000+0000",
        "quarantine_setting_id": 1,
        "dataSetId": null,
        "size": 3033,
        "updated_at": "2018-08-23T03:02:34.000+0000",
        "cron_frequency": "0 0 0 * * ? *",
        "org_id": 1,
        "name": "Sundeep/test/source/5002/2018/8/22-5ba76dd7-7e6a-490a-8503-fbbf973defa1-ingest.json",
        "id": 1,
        "data_credentials_id": 5010,
        "resourceType": "SOURCE"
      }
    ],
    "meta": {
      "currentPage": 1,
      "totalCount": 1,
      "pageCount": 1
    }
  }
}

Audit Logs​

In order to support full traceability of issues, Nexla records any changes made to any property of a resource by a user or by the platform. You can fetch detailed audit history for any resource you have access to by calling the audit_log endpoint for that resource.

For example, the call below fetches audit history for data source id 9834. Replace /data_sources/{data_source_id} with /data_sets/{data_set_id}, /data_sinks/{data_sink_id},/data_credentials/{data_credential_id}, /data_maps/{data_map_id}, /code_containers/{transform_id} or /data_schemas/{data_schema_id}.

Audit Log: Request
GET /{resource_type}/{resource_id}/audit_log
...
Example: /data_sources/9834/audit_log

The object_changes in each audit event contains info about what properties where changed, and the before and after values. For example, the first entry in this sample response says status of source 9834 was changed from ACTIVE to PAUSED at UTC time 2019-01-15 16:41:36

Audit Log: Response
[
  {
    "id": 8328,
    "item_type": "DataSource",
    "item_id": 9834,
    "event": "update",
    "org_id": 22,
    "impersonator_id": null,
    "owner_id": 119,
    "owner_email": "demo_user@testnexla.com",
    "request_ip": "130.211.2.53",
    "request_user_agent": "mozilla/5.0 (macintosh; intel mac os x 10_14_0) applewebkit/537.36 (khtml, like gecko) chrome/71.0.3578.98 safari/537.36",
    "user": {
      "id": 119,
      "email": "demo_user@testnexla.com"
    },
    "owner": {
      "id": 119,
      "full_name": "Demo User",
      "email": "demo_user@testnexla.com",
      "email_verified_at": "2019-05-28T19:13:43.000Z"
    },
    "object_changes": {
      "status": ["ACTIVE", "PAUSED"],
      "updated_at": ["2019-01-15 15:30:18", "2019-01-15 16:41:36"]
    },
    "created_at": "2019-01-15T16:41:36.254Z"
  },
  {
    "id": 8321,
    "item_type": "DataSource",
    "item_id": 9834,
    "event": "update",
    "org_id": 22,
    "impersonator_id": null,
    "owner_id": 119,
    "owner_email": "demo_user@testnexla.com",
    "request_ip": "130.211.2.153",
    "request_user_agent": "mozilla/5.0 (macintosh; intel mac os x 10_14_0) applewebkit/537.36 (khtml, like gecko) chrome/71.0.3578.98 safari/537.36",
    "user": {
      "id": 119,
      "email": "demo_user@testnexla.com"
    },
    "owner": {
      "id": 119,
      "full_name": "Demo User",
      "email": "demo_user@testnexla.com",
      "email_verified_at": "2019-05-28T19:13:43.000Z"
    },
    "object_changes": {
      "status": ["INIT", "ACTIVE"],
      "updated_at": ["2019-01-15 15:29:50", "2019-01-15 15:30:18"]
    },
    "created_at": "2019-01-15T15:30:18.880Z"
  },
  {
    "id": 8320,
    "item_type": "DataSource",
    "item_id": 9834,
    "event": "create",
    "org_id": 22,
    "impersonator_id": null,
    "owner_id": 119,
    "owner_email": "demo_user@testnexla.com",
    "request_ip": "130.211.2.153",
    "request_user_agent": "mozilla/5.0 (macintosh; intel mac os x 10_14_0) applewebkit/537.36 (khtml, like gecko) chrome/71.0.3578.98 safari/537.36",
    "user": {
      "id": 119,
      "email": "demo_user@testnexla.com"
    },
    "owner": {
      "id": 119,
      "full_name": "Demo User",
      "email": "demo_user@testnexla.com",
      "email_verified_at": "2019-05-28T19:13:43.000Z"
    },
    "object_changes": {
      "status": [null, "INIT"],
      "updated_at": [null, "2019-01-15 15:29:50"],
      "owner_id": [null, 119],
      "org_id": [null, 22],
      "ingest_method": [null, "API"],
      "source_format": [null, "JSON"],
      "name": [null, "Test Webhook"],
      "source_type": ["s3", "nexla_rest"],
      "source_config": [
        null,
        {
          "schema.detection.once": true,
          "webhook.auth_type": "Nexla API Key",
          "hash.enabled": false
        }
      ],
      "created_at": [null, "2019-01-15 15:29:50"],
      "id": [null, 9834]
    },
    "created_at": "2019-01-15T15:29:50.272Z"
  }
]