Codefresh Data Source

The Codefresh connector enables you to ingest CI/CD data from your Codefresh account, including pipelines, pipeline builds (workflow runs), projects, triggers, integration and execution contexts, agents, runtime environments, audit logs, and analytics reports. This connector is particularly useful for teams that need to monitor build activity, report on pipeline performance, or sync Codefresh configuration data into downstream systems. Follow the instructions below to create a new data flow that ingests data from a Codefresh source in Nexla.

Codefresh

Create a New Data Flow

1. To create a new data flow, navigate to the Integrate section, and click the New Data Flow button. Then, select the desired flow type from the list, and click the Create button.

2. Select the Codefresh connector tile from the list of available connectors. Then, select the credential that will be used to connect to the Codefresh instance, and click Next; or, create a new Codefresh credential for use in this flow.

3. In Nexla, Codefresh data sources can be created using pre-built endpoint templates, which expedite source setup for common Codefresh endpoints. Each template is designed specifically for the corresponding Codefresh endpoint, making source configuration easy and efficient.
   • To configure this source using a template, follow the instructions in Configure Using a Template.

   Codefresh sources can also be configured manually, allowing you to ingest data from Codefresh endpoints not included in the pre-built templates or apply further customizations to exactly suit your needs.
   • To configure this source manually, follow the instructions in Configure Manually.

Configure Using a Template

Nexla provides pre-built templates that can be used to rapidly configure data sources to ingest data from common Codefresh endpoints. Each template is designed specifically for the corresponding Codefresh endpoint, making data source setup easy and efficient.

Endpoint Settings

• Select the endpoint from which this source will fetch data from the Endpoint pulldown menu. Available endpoint templates are listed in the expandable boxes below. Click on an endpoint to see more information about it and how to configure your data source for this endpoint.

  List Pipelines

  This endpoint returns a list of all pipelines in your Codefresh account. Use it to inventory your pipelines, report on pipeline configurations, or feed pipeline metadata into downstream systems.

  • This endpoint requires no additional configuration. Select it from the Endpoint menu to retrieve all pipelines accessible to your credential.
  • Results are paginated automatically. Nexla pages through the results using offset-based pagination, fetching up to 100 records per page until all pipelines have been retrieved.

  In Codefresh, pipelines are workflows composed of individual steps, and they are grouped within projects. The response includes each pipeline's metadata and specification.

  List Pipeline Builds/Runs

  This endpoint returns a list of workflow builds (pipeline runs) in your account. Use it to monitor build activity, analyze build outcomes and durations, or report on CI/CD throughput.

  • This endpoint requires no additional configuration. Select it from the Endpoint menu to retrieve recent pipeline builds.
  • Results are paginated automatically using page-based pagination, fetching up to 100 builds per page until no further data is returned.

  A build (also called a workflow run) represents a single execution of a pipeline. Each build record includes status, timing, and trigger information that is useful for build analytics.

  Get Pipeline Build/Run

  This endpoint retrieves detailed status and metadata for a single pipeline build (workflow run) by its ID. Use it when you need the full detail of one specific run rather than a summary list.

  • Enter the build identifier in the Build ID field. This is the workflow/build ID of the pipeline run to retrieve. You can obtain build IDs from the "List Pipeline Builds/Runs" endpoint.

  Build IDs are unique identifiers for individual workflow runs. Use the "List Pipeline Builds/Runs" endpoint first to find the ID of the run you want to inspect.

  List Projects

  This endpoint returns a list of projects in your account. In Codefresh, a project acts as a folder that groups related pipelines, such as all pipelines for the microservices of a single application.

  • This endpoint requires no additional configuration. Select it from the Endpoint menu to retrieve all projects.
  • Results are paginated automatically using offset-based pagination, fetching up to 100 projects per page until all projects have been retrieved.

  Project records are useful for understanding how pipelines are organized and for mapping pipelines to applications or teams.

  List Pipeline Triggers

  This endpoint returns a list of pipeline triggers (Git, Cron, Helm, and others) configured across your account. Use it to audit how and when pipelines are automatically initiated.

  • This endpoint requires no additional configuration. Select it from the Endpoint menu to retrieve all configured triggers.

  Triggers define the events that start a pipeline, such as a Git push, a scheduled Cron expression, or a Helm chart update.

  List Step Types

  This endpoint returns a list of available step types that can be used in Codefresh pipelines. Each step in a pipeline performs a specific action, and step types are the reusable building blocks for those steps.

  • This endpoint requires no additional configuration. Select it from the Endpoint menu to retrieve the catalog of available step types.
  • Results are paginated automatically using offset-based pagination, fetching up to 100 records per page until all step types have been retrieved.

  Step types include both built-in steps provided by Codefresh and custom steps published to the step marketplace.

  List Contexts

  This endpoint returns a list of contexts configured in your account. Contexts store shared configuration and integration details that pipelines reference at runtime.

  • This endpoint requires no additional configuration. Select it from the Endpoint menu to retrieve all contexts.

  Use the "List Integration Contexts" workflow on the destination side to manage contexts such as Git providers, registries, and cloud providers.

  List Execution Contexts

  This endpoint returns a list of execution contexts available in your account. Execution contexts define where and how pipeline steps run.

  • This endpoint requires no additional configuration. Select it from the Endpoint menu to retrieve all execution contexts.

  Execution contexts are useful for understanding the runtime configuration applied to pipeline executions.

  List Runtime Environments

  This endpoint returns a list of available runtime environments (Kubernetes-based runners) configured for your account. Runtime environments are the Kubernetes clusters on which pipeline builds execute.

  • This endpoint requires no additional configuration. Select it from the Endpoint menu to retrieve all runtime environments.

  Runtime environment data helps you understand the compute infrastructure backing your pipelines and is useful for capacity and cost reporting.

  List Agents

  This endpoint returns a list of agents configured in your account. Agents (Codefresh Runner agents) connect your Kubernetes clusters to the Codefresh platform.

  • This endpoint requires no additional configuration. Select it from the Endpoint menu to retrieve all agents.

  Agent records are useful for monitoring the health and connectivity of your self-hosted Codefresh runners.

  List Helm Repositories

  This endpoint returns a list of Helm repositories configured in your Codefresh account. Use it to inventory the Helm chart sources available to your pipelines.

  • This endpoint requires no additional configuration. Select it from the Endpoint menu to retrieve all Helm repositories.

  Helm repository data is useful for teams that deploy applications to Kubernetes using Helm charts as part of their CD process.

  List Account Users

  This endpoint returns a list of users associated with your account. Use it for access auditing, user reporting, or syncing account membership into other systems.

  • This endpoint requires no additional configuration. Select it from the Endpoint menu to retrieve all account users.

  Retrieving account users typically requires a key with administrative scopes. Ensure your API key has the appropriate access for account-level data.

  Get Account Settings

  This endpoint retrieves the settings for the current account. Use it to capture account-level configuration for reporting or compliance purposes.

  • This endpoint requires no additional configuration. Select it from the Endpoint menu to retrieve the current account settings.

  Account settings reflect the configuration of the account associated with your API key.

  List Audit Logs

  This endpoint returns a list of audit logs for account activities. Use it to track changes and actions performed in your account for security and compliance reporting.

  • This endpoint requires no additional configuration. Select it from the Endpoint menu to retrieve audit log entries.

  Audit log access typically requires administrative scopes. Audit data is valuable for security monitoring and demonstrating compliance.

  List Analytics Reports

  This endpoint returns a list of available analytics reports. Use it to discover the reports you can pull metrics from for build and pipeline performance analysis.

  • This endpoint requires no additional configuration. Select it from the Endpoint menu to retrieve the list of analytics reports.

  Pair this endpoint with "Get Analytics Metadata" to understand the structure and available fields of each report.

  Get Analytics Metadata

  This endpoint retrieves metadata for analytics data, describing the structure and available dimensions of Codefresh analytics reports.

  • This endpoint requires no additional configuration. Select it from the Endpoint menu to retrieve analytics metadata.

  Analytics metadata is helpful for interpreting the fields returned by the "List Analytics Reports" endpoint before ingesting report data.

Endpoint Testing

Once the selected endpoint template has been configured, Nexla can retrieve a sample of the data that will be fetched according to the current settings. This allows users to verify that the source is configured correctly before saving.

• To test the current endpoint configuration, click the Test button to the right of the endpoint selection menu. Sample data will be fetched & displayed in the Endpoint Test Result panel on the right.

• If the sample data is not as expected, review the selected endpoint and associated settings, and make any necessary adjustments. Then, click the Test button again, and check the sample data to ensure that the correct information is displayed.

Configure Manually

Codefresh data sources can be manually configured to ingest data from any valid Codefresh API endpoint. Manual configuration provides maximum flexibility for accessing endpoints not covered by pre-built templates or when you need custom API configurations.

With manual configuration, you can also create more complex Codefresh sources, such as sources that use chained API calls to fetch data from multiple endpoints or sources that require custom request parameters.

API Method

1. To manually configure this source, select the Advanced tab at the top of the configuration screen.

2. Select the API method that will be used for calls to the Codefresh API from the Method pulldown menu. Most Codefresh read operations use:

   • GET: For retrieving data from the API

API Endpoint URL

3. Enter the URL of the Codefresh API endpoint from which this source will fetch data in the Set API URL field. This should be the complete URL including the protocol (https://) and any required path parameters. For example, the pipelines endpoint on the Codefresh SaaS platform is https://g.codefresh.io/api/pipelines.

Ensure the API endpoint URL is correct and accessible with your current credentials. You can test the endpoint using the Test button after configuring the URL.

Path to Data

Optional

If only a subset of the data returned by the API endpoint is needed, you can designate the part(s) of the response that should be included in the Nexset(s) produced from this source by specifying the path to the relevant data within the response. This is particularly useful because Codefresh wraps lists in container objects—for example, pipelines and builds are returned inside docs arrays, and users are returned inside a users array.

For example, when fetching a list of pipelines, the API returns the pipeline records inside a docs array. By entering the path to that array, you can configure Nexla to treat each element as a record.

Path to Data is essential when API responses have nested structures. Without specifying the correct path, Nexla might not be able to properly parse and organize your data into usable records.

• To specify which data should be treated as relevant in responses from this source, enter the path to the relevant data in the Set Path to Data in Response field. Codefresh responses are in JSON format, so enter the JSON path that points to the object or array that should be treated as relevant data. JSON paths use dot notation.

  Path to Data Examples:

  For the pipelines and builds endpoints, the relevant records are nested in a docs array, so the path would be entered as $.docs[*]. For the projects endpoint, use $.projects[*], and for the account users endpoint, use $.users[*].

Autogenerate Path Suggestions

Nexla can also autogenerate data path suggestions based on the response from the API endpoint. These suggested paths can be used as-is or modified to exactly suit your needs.

• To use this feature, click the Test button next to the Set API URL field to fetch a sample response from the API endpoint. Suggested data paths generated based on the content & format of the response will be displayed in the Suggestions box below the Set Path to Data in Response field.

• Click on a suggestion to automatically populate the Set Path to Data in Response field with the corresponding path. The populated path can be modified directly within the field if further customization is needed.

  

Metadata

If metadata is included in the response but is located outside of the defined path to relevant data, you can configure Nexla to include this data as common metadata in each record. This is useful when you want to preserve important contextual information that applies to all records but isn't part of the main data array.

For example, Codefresh list responses often include pagination metadata alongside the docs array. In this case, if you have specified the path to the relevant data but want to retain pagination or summary metadata, you can specify a path to this metadata to include it with each record in the generated Nexset(s).

Metadata paths are particularly useful for preserving API response context like total counts or pagination information that applies to all records in the response.

• To specify the location of metadata that should be included with each record, enter the JSON path to the relevant metadata in the Path to Metadata in Response field.

Request Headers

Optional

• If Nexla should include any additional request headers in API calls to this source, enter the headers & corresponding values as comma-separated pairs in the Request Headers field (e.g., header1:value1,header2:value2). Additional headers are often required for API versioning, content type specifications, or custom requirements.

  You do not need to include any headers already present in the credentials. The Authorization header containing your Codefresh API key is added automatically by Nexla based on your credential configuration.

Endpoint Testing

After configuring all settings for the selected endpoint, Nexla can retrieve a sample of the data that will be fetched according to the current configuration. This allows users to verify that the source is configured correctly before saving.

• To test the current endpoint configuration, click the Test button to the right of the endpoint selection menu. Sample data will be fetched & displayed in the Endpoint Test Result panel on the right.

• If the sample data is not as expected, review the selected endpoint and associated settings, and make any necessary adjustments. Then, click the Test button again, and check the sample data to ensure that the correct information is displayed.

Save & Activate the Source

1. Once all of the relevant steps in the above sections have been completed, click the Create button in the upper right corner of the screen to save and create the new Codefresh data source. Nexla will now begin ingesting data from the configured endpoint and will organize any data that it finds into one or more Nexsets.