Skip to main content

Data Source

Follow the instructions below to create a new data flow that ingests data from a DataSet / Scalyr source in Nexla.
scalyr_api.png

DataSet / Scalyr

Create a New Data Flow​

  1. To create a new data flow, navigate to the Integrate section, and click the New Data Flow button. Then, select the desired flow type from the list, and click the Create button.

  2. Select the DataSet / Scalyr connector tile from the list of available connectors. Then, select the credential that will be used to connect to your DataSet / Scalyr account, and click Next; or, create a new DataSet / Scalyr credential for use in this flow.

  3. In Nexla, DataSet / Scalyr data sources can be created using pre-built endpoint templates, which expedite source setup for common DataSet / Scalyr endpoints. Each template is designed specifically for the corresponding DataSet / Scalyr endpoint, making source configuration easy and efficient.
    • To configure this source using a template, follow the instructions in Configure Using a Template.

    DataSet / Scalyr sources can also be configured manually, allowing you to ingest data from DataSet / Scalyr endpoints not included in the pre-built templates or apply further customizations to exactly suit your needs.
    • To configure this source manually, follow the instructions in Configure Manually.

Configure Using a Template​

Nexla provides pre-built templates that can be used to rapidly configure data sources to ingest data from common DataSet / Scalyr endpoints. Each template is designed specifically for the corresponding DataSet / Scalyr endpoint, making data source setup easy and efficient.

Endpoint Settings​

  • Select the endpoint from which this source will fetch data from the Endpoint pulldown menu. Available endpoint templates are listed in the expandable boxes below. Click on an endpoint to see more information about it and how to configure your data source for this endpoint.

    Execute a Query

    This endpoint template retrieves events (log records) from your DataSet / Scalyr account. You can retrieve all events in a specified time range, or only events matching specific criteria. Use this template when you need to query log data from your DataSet / Scalyr account.

    • Enter the query payload in the Query Payload (JSON Object) field. This should be a valid JSON object that includes a key queryType with value log. The query payload determines which events will be retrieved. You can include additional query parameters such as startTime, endTime, and filter criteria. Example: `{"queryType": "log", "startTime": "{now-1}"}`
    • Select the date format from the Date Format pulldown menu. Available options include Unix Epoch Time (Seconds) (default), Unix Epoch Time (Milliseconds), ISO-8601 date format, yyyy-MM-dd, and other date formats. The date format determines how dates will be formatted in the query payload and response.

    This endpoint returns events (log records) from your DataSet / Scalyr account based on the query payload. The endpoint uses continuation token-based pagination with continuationToken to handle large datasets efficiently. Nexla will automatically fetch subsequent pages of data by following the continuation token returned in the API response.

    The query payload must be a valid JSON object and must include a key queryType with value log. You can include additional query parameters such as startTime, endTime, and filter criteria to customize the data retrieval. For detailed information about query syntax, API response structures, pagination, and available query parameters, see the DataSet / Scalyr API documentation.

    List Users

    This endpoint template lists all users in your DataSet / Scalyr account. Use this template when you need to retrieve information about users, including user IDs, names, and other user metadata.

    • This endpoint automatically retrieves all users from your DataSet / Scalyr account. No additional configuration is required beyond selecting this endpoint template.

    This endpoint returns all users in your DataSet / Scalyr account, including user IDs, names, and other user metadata. Use this endpoint to discover which users are available in your DataSet / Scalyr account.

    For detailed information about user management, API response structures, and available user data, see the DataSet / Scalyr API documentation.

Endpoint Testing​

Once the selected endpoint template has been configured, Nexla can retrieve a sample of the data that will be fetched according to the current settings. This allows users to verify that the source is configured correctly before saving.

  • To test the current endpoint configuration, click the Test button to the right of the endpoint selection menu. Sample data will be fetched & displayed in the Endpoint Test Result panel on the right.

  • If the sample data is not as expected, review the selected endpoint and associated settings, and make any necessary adjustments. Then, click the Test button again, and check the sample data to ensure that the correct information is displayed.

Configure Manually​

DataSet / Scalyr data sources can be manually configured to ingest data from any valid DataSet / Scalyr API endpoint. Manual configuration provides maximum flexibility for accessing endpoints not covered by pre-built templates or when you need custom API configurations.

With manual configuration, you can also create more complex DataSet / Scalyr sources, such as sources that use chained API calls to fetch related data or sources that require custom query parameters or filters.

API Method​

  1. To manually configure this source, select the Advanced tab at the top of the configuration screen.

  2. Select the API method that will be used for calls to the DataSet / Scalyr API from the Method pulldown menu. DataSet / Scalyr API typically uses POST method for query endpoints and GET method for other endpoints.

API Endpoint URL​

  1. Enter the URL of the DataSet / Scalyr API endpoint from which this source will fetch data in the Set API URL field. This should be the complete URL to your DataSet / Scalyr endpoint (e.g., https://app.scalyr.com/api/query, https://app.scalyr.com/api/listUsers). Include any required query parameters in the URL.

Ensure the API endpoint URL is correct and accessible with your current credentials. The DataSet / Scalyr API endpoint URL should include the API path (/api/) and the resource type path. You can test the endpoint using the Test button after configuring the URL.

Response Data Path​

  1. Enter the JSONPath expression in the Response Data Path field to specify which part of the API response should be treated as the relevant data by Nexla. For DataSet / Scalyr API responses, use $.matches[*] to extract all matches from the matches array for query endpoints, $.users[*] to extract all users from the users array for user endpoints, or $ to extract the entire response for single record endpoints, depending on your endpoint.

The JSONPath expression must correctly reference the structure of your DataSet / Scalyr API response. DataSet / Scalyr API responses may have different structures depending on the endpoint. Ensure your JSONPath expression matches the structure returned by your specific endpoint. The JSONPath expression determines which data will be extracted and processed by Nexla.

Pagination (if applicable)​

  1. If your endpoint supports pagination, configure the pagination settings in the Pagination section. DataSet / Scalyr API uses continuation token-based pagination with continuationToken for query endpoints. Select the pagination type that matches your endpoint's pagination mechanism.

  2. Configure the pagination parameters based on your selected pagination type. For continuation token-based pagination, specify the token parameter name (continuationToken), the JSONPath expression to the continuation token in the response ($.continuationToken), and indicate that the token parameter is included in the request body (params.in.body: true).

Save the Data Source​

  1. Once all of the relevant steps in the above sections have been completed, click the Next button to proceed with the rest of the data flow configuration, or click Save to save the data source configuration for later use.