Skip to main content

Buildkite Data Source

The Buildkite connector enables you to ingest CI/CD pipeline data — including builds, pipelines, agents, jobs, artifacts, and test analytics — directly into Nexla for reporting, monitoring, and downstream analysis. Follow the instructions below to create a new data flow that ingests data from a Buildkite source in Nexla.
buildkite_api.png

Buildkite

Create a New Data Flow​

  1. To create a new data flow, navigate to the Integrate section, and click the New Data Flow button. Then, select the desired flow type from the list, and click the Create button.

  2. Select the Buildkite connector tile from the list of available connectors. Then, select the credential that will be used to connect to the Buildkite instance, and click Next; or, create a new Buildkite credential for use in this flow.

  3. In Nexla, Buildkite data sources can be created using pre-built endpoint templates, which expedite source setup for common Buildkite endpoints. Each template is designed specifically for the corresponding Buildkite endpoint, making data source setup easy and efficient.
    • To configure this source using a template, follow the instructions in Configure Using a Template.

    Buildkite sources can also be configured manually, allowing you to ingest data from Buildkite endpoints not included in the pre-built templates or apply further customizations to exactly suit your needs.
    • To configure this source manually, follow the instructions in Configure Manually.

Configure Using a Template​

Nexla provides pre-built templates that can be used to rapidly configure data sources to ingest data from common Buildkite endpoints. Each template is designed specifically for the corresponding Buildkite endpoint, making data source setup easy and efficient.

Endpoint Settings​

  • Select the endpoint from which this source will fetch data from the Endpoint pulldown menu. Available endpoint templates are listed in the expandable boxes below. Click on an endpoint to see more information about it and how to configure your data source for this endpoint.

    List Organizations

    Returns a list of all organizations accessible to the authenticated user. Use this endpoint to enumerate the Buildkite organizations your API token has access to.

    Supports pagination via page and per_page query parameters (maximum 100 results per page). This endpoint requires a valid API access token with organization read permissions.

    List Test Suites Analytics

    Returns analytics data for test suites within an organization. Use this endpoint to monitor test suite health and track trends in test execution across your CI/CD pipelines.

    Test Analytics must be enabled for your Buildkite organization and plan. The organization slug is case-sensitive and must match the slug shown in your Buildkite dashboard URL.

    List Organization Pipelines

    Returns a list of all pipelines in an organization. Use this endpoint to retrieve pipeline configurations and metadata for all CI/CD pipelines defined in a Buildkite organization.

    Pagination is supported via page and per_page query parameters. Results include pipeline settings, steps, and repository information.

    Get Current Access Token

    Returns details about the current access token and its associated permissions. Use this endpoint to verify which scopes and organizations your API token can access.

    This endpoint is useful for validating credential configuration and diagnosing permission issues before setting up other endpoints.

    List All Builds

    Returns a list of all builds across all organizations and pipelines accessible to the authenticated token. Use this endpoint for a unified view of build activity across your entire Buildkite account.

    Supports filtering by state, branch, and date range using query parameters such as state, branch, created_from, and finished_from. Use date/time macros in the URL to implement incremental ingestion.

    List Organization Clusters

    Returns a list of all clusters in an organization. Use this endpoint to retrieve information about agent clusters configured for your Buildkite organization.

    Clusters are a Buildkite feature for grouping agents. This endpoint requires the Clusters feature to be enabled on your Buildkite plan.

    List Organization Builds

    Returns a list of all builds in an organization. Use this endpoint to retrieve build history for all pipelines within a specific Buildkite organization.

    Supports filtering by build state, branch, and date range. Use date/time macros with created_from or finished_from query parameters for incremental data ingestion.

    List Pipeline Builds

    Returns a list of all builds in a specific pipeline. Use this endpoint when you need build history scoped to a single pipeline rather than across the entire organization.

    The pipeline slug is found in the Buildkite dashboard URL when viewing the pipeline, e.g., https://buildkite.com/my-org/my-pipeline → pipeline slug is my-pipeline.

    List Cluster Queues

    Returns a list of all queues in a cluster. Use this endpoint to retrieve agent queue definitions and configurations within a specific Buildkite cluster.

    Cluster IDs can be retrieved using the List Organization Clusters endpoint. This endpoint requires the Clusters feature to be enabled on your Buildkite plan.

    List Cluster Tokens

    Returns a list of all tokens associated with a cluster. Use this endpoint to audit agent tokens configured for a specific Buildkite cluster.

    Token values themselves are redacted in API responses for security. This endpoint returns token metadata such as description and allowed IP addresses.

    List Organization Emojis

    Returns a list of custom emojis available in the organization. Use this endpoint to retrieve the custom emoji catalog defined for your Buildkite organization.

    This endpoint returns both built-in and custom emojis. Custom emojis are defined at the organization level in Buildkite settings.

    Get Current User

    Returns the current authenticated user's profile information. Use this endpoint to retrieve the Buildkite user account associated with the API token in use.

    This endpoint returns the user associated with the API token, including name, email, and avatar URL. It is useful for validating which user account is associated with the configured credential.

Endpoint Testing​

Once the selected endpoint template has been configured, Nexla can retrieve a sample of the data that will be fetched according to the current settings. This allows users to verify that the source is configured correctly before saving.

  • To test the current endpoint configuration, click the Test button to the right of the endpoint selection menu. Sample data will be fetched & displayed in the Endpoint Test Result panel on the right.

  • If the sample data is not as expected, review the selected endpoint and associated settings, and make any necessary adjustments. Then, click the Test button again, and check the sample data to ensure that the correct information is displayed.

Configure Manually​

Buildkite sources can also be configured manually, allowing you to ingest data from any valid Buildkite REST API endpoint. The Buildkite REST API is available at https://api.buildkite.com/v2 and provides access to builds, pipelines, organizations, jobs, agents, artifacts, and more. Configuration options allow sources to be fully customized to suit any use case — including using chained API calls to fetch data from multiple endpoints.

First, select the API method that will be used for calls to the Buildkite API from the Method pulldown menu. The most common methods are:

  • GET: For retrieving data from the API — the primary method for ingesting Buildkite data into Nexla
  • POST: For sending data to the API or triggering actions, such as creating a new build
  • PUT: For updating existing data
  • PATCH: For partial updates to existing data
  • DELETE: For removing data

API Endpoint URL​

  1. Enter the URL of the Buildkite API endpoint from which this source will fetch data in the Set API URL field. This should be the complete URL including the protocol and any required path parameters.

    Common Buildkite REST API endpoints include:

    • List all builds for an organization: https://api.buildkite.com/v2/organizations/{'{org_slug}'}/builds
    • List builds for a pipeline: https://api.buildkite.com/v2/organizations/{'{org_slug}'}/pipelines/{'{pipeline_slug}'}/builds
    • List all pipelines: https://api.buildkite.com/v2/organizations/{'{org_slug}'}/pipelines
    • List all agents: https://api.buildkite.com/v2/organizations/{'{org_slug}'}/agents
    • List artifacts for a build: https://api.buildkite.com/v2/organizations/{'{org_slug}'}/pipelines/{'{pipeline_slug}'}/builds/{'{build_number}'}/artifacts

    Replace {'{org_slug}'} with your Buildkite organization slug, which can be found in the URL of your Buildkite organization dashboard (for example, https://buildkite.com/my-org → slug is my-org).

The Buildkite REST API base URL is https://api.buildkite.com/v2. All endpoints are accessed over HTTPS. The API supports pagination using page and per_page query parameters, with a maximum of 100 results per page.

Date/Time Macros (API URL)​

Optional

Optionally, the API URL can be customized using macros — all macros added to the API URL will be converted into values when Nexla executes the API call. Macros are dynamic placeholders that allow you to create flexible API endpoints that can adapt to different time periods or data requirements.

This is particularly useful for Buildkite endpoints that support date filtering, such as filtering builds by creation time or completion time.

Macros are particularly useful for Buildkite APIs that accept date range filters, such as created_from, created_to, finished_from, or finished_to query parameters on the Builds API endpoint.

  1. To add a macro, type { at the appropriate position in the API URL (within the Set API URL field), and select the desired macro from the dropdown list.

    • {now} – The current datetime
    • {now-1} – The datetime one time unit before the current datetime
    • {now+1} – The datetime one time unit after the current datetime
    • custom – Datetime macros can reference any number of time units before or after the current datetime — for example, enter (now-4) to indicate the datetime four time units before the current datetime

  2. Select the format that will be applied to datetime macros from the Date Format for Date/Time Macro pulldown menu. This format will be applied to the base datetime value of the macro — i.e., the value of {now} in {now-1}. The Buildkite API expects dates in ISO 8601 format (e.g., 2024-01-15T00:00:00Z).

  3. Select the datetime unit that will be used to perform mathematical operations in the included macro(s) from the Time Unit for Operations pulldown menu — for example, for the macro {now-1}, when Day is selected, {now-1} will be converted to the datetime one day before the current datetime.

Lookup-Based Macros (API URL)​

Optional

Column values from existing lookups can also be included as macros in the API URL. Lookup-based macros allow you to reference data from previously configured data sources or lookups, enabling dynamic API endpoints that can adapt based on existing data.

Lookup-based macros are useful when you need to dynamically pass Buildkite identifiers — such as organization slugs, pipeline slugs, or build numbers — from another Nexla data source into the API URL.

  1. To include a lookup column value macro, select the relevant lookup from the Add Lookups to Supported Macros pulldown menu.

  2. Type { at the appropriate position in the API URL, and select the lookup column-based macro from the dropdown list. Lookup-based macros are automatically populated into the macro list when a lookup is selected in the Add Lookups to Supported Macros pulldown menu.

Path to Data​

Optional

If only a subset of the data returned by the Buildkite API endpoint is needed, you can designate the part of the response that should be included in the Nexset(s) produced from this source by specifying the path to the relevant data within the response.

Most Buildkite REST API list endpoints return JSON arrays at the top level of the response. For example, the Builds API returns an array of build objects directly. In some cases, additional metadata may accompany the main data array, and specifying a data path ensures Nexla correctly identifies the records to ingest.

Path to Data is essential when API responses have nested structures. Without specifying the correct path, Nexla might not be able to properly parse and organize your data into usable records.

  • To specify which data should be treated as relevant in responses from this source, enter the path to the relevant data in the Set Path to Data in Response field.

    • For responses in JSON format, enter the JSON path that points to the object or array that should be treated as relevant data. JSON paths use dot notation (e.g., $.data.items[*] to access an array of items within a data object).

    • For responses in XML format, enter the XPath that points to the object/array containing relevant data. XPath uses slash notation (e.g., /response/data/item to access item elements within a data element).

    Path to Data Example:

    The Buildkite Builds API returns a JSON array of build objects at the top level of the response. If the response is wrapped in an object (as in some filtered requests), the path to the array of builds would be entered as $[*] for top-level arrays or $.builds[*] if nested under a key.

Autogenerate Path Suggestions​

Nexla can also autogenerate data path suggestions based on the response from the API endpoint. These suggested paths can be used as-is or modified to exactly suit your needs.

  • To use this feature, click the Test button next to the Set API URL field to fetch a sample response from the API endpoint. Suggested data paths generated based on the content and format of the response will be displayed in the Suggestions box below the Set Path to Data in Response field.

  • Click on a suggestion to automatically populate the Set Path to Data in Response field with the corresponding path. The populated path can be modified directly within the field if further customization is needed.

    PathSuggestions.png

Metadata​

If metadata is included in the response but is located outside of the defined path to relevant data, you can configure Nexla to include this data as common metadata in each record.

For example, Buildkite API responses may include pagination metadata such as the total number of records or a cursor for the next page. If you have specified the path to the array of build records, you can specify a separate path to preserve this pagination or request metadata alongside each record.

Metadata paths are particularly useful for preserving Buildkite API response context like request IDs, timestamps, or pagination counts that apply to all records in the response.

  • To specify the location of metadata that should be included with each record, enter the path to the relevant metadata in the Path to Metadata in Response field.

    • For responses in JSON format, enter the JSON path to the object or array that contains the metadata, and for responses in XML format, enter the XPath.

Request Headers​

Optional

  • If Nexla should include any additional request headers in API calls to this source, enter the headers and corresponding values as comma-separated pairs in the Request Headers field (e.g., header1:value1,header2:value2). Additional headers are useful for specifying API versioning requirements or content type expectations.

    You do not need to include the Authorization header, as it is handled automatically by Nexla using the API key stored in your Buildkite credential. Common headers like Accept: application/json are also handled automatically.

Buildkite API Rate Limits

The Buildkite REST API enforces rate limits with a 60-second window. Rate limit status is returned in response headers: RateLimit-Remaining, RateLimit-Limit, and RateLimit-Reset. If your Nexla data flow encounters rate limit errors, consider increasing the polling interval or reducing the scope of each API request using pagination parameters (page and per_page, with a maximum of 100 results per page).

Save & Activate the Source​

  1. Once all of the relevant steps in the above sections have been completed, click the Create button in the upper right corner of the screen to save and create the new Buildkite data source. Nexla will now begin ingesting data from the configured endpoint and will organize any data that it finds into one or more Nexsets.