Skip to main content

Google BigQuery API Data Source

Nexla's bi-directional connectors allow data to flow both to and from any location, making it simple to create a FlexFlow, DB-CDC, Spark ETL, or Replication data flow that ingests data from a Google BigQuery API location.
gbq_api.png

Google BigQuery API

Follow the instructions below to create a new data flow that ingests data from a Google BigQuery API source in Nexla.

Create a New Data Flow​

  1. To create a new data flow, navigate to the Integrate section, and click the New Data Flow button. Then, select the desired flow type from the list, and click the Create button.

  2. Select the Google BigQuery API connector tile from the list of available connectors. Then, select the credential that will be used to connect to the Google BigQuery API, and click Next; or, create a new Google BigQuery API credential for use in this flow.

  3. In Nexla, Google BigQuery API data sources can be created using pre-built endpoint templates, which expedite source setup for common Google BigQuery API endpoints. Each template is designed specifically for the corresponding Google BigQuery API endpoint, making source configuration easy and efficient.
    • To configure this source using a template, follow the instructions in Configure Using a Template.

    Google BigQuery API sources can also be configured manually, allowing you to ingest data from Google BigQuery API endpoints not included in the pre-built templates or apply further customizations to exactly suit your needs.
    • To configure this source manually, follow the instructions in Configure Manually.

Configure Using a Template​

Nexla provides pre-built templates that can be used to rapidly configure data sources to ingest data from common Google BigQuery API endpoints. Each template is designed specifically for the corresponding Google BigQuery API endpoint, making data source setup easy and efficient.

Endpoint Settings​

  • Select the endpoint from which this source will fetch data from the Endpoint pulldown menu. Available endpoint templates are listed in the expandable boxes below. Click on an endpoint to see more information about it and how to configure your data source for this endpoint.

    List all Projects

    List projects to which the user has been granted any project role.

    • This endpoint automatically lists all Google Cloud projects to which your authenticated account has been granted any project role. No additional configuration is required beyond selecting this endpoint template.
    • The endpoint uses GET requests to https://bigquery.googleapis.com/bigquery/v2/projects to retrieve a list of all projects accessible to your account. The endpoint URL is automatically constructed based on the Google BigQuery API base URL.
    • The endpoint uses pagination (iteration.type: paging.next.token) to retrieve all projects. Pagination uses the nextPageToken from the API response to navigate through pages. The pageToken query parameter is used to request the next page of results. The endpoint will automatically fetch all pages until all projects are retrieved.
    • The response data path is $.projects[*], which extracts all project objects from the projects array in the API response. The next page token is extracted from $.nextPageToken for pagination.
    • The response format is JSON, which is automatically parsed by Nexla.

    This endpoint retrieves a list of all Google Cloud projects to which your authenticated account has been granted any project role. The endpoint requires OAuth2 authentication via the Authorization: Bearer {token} header, which is handled automatically by your credential configuration. The endpoint uses token-based pagination to retrieve all projects across multiple pages. The projects returned include project metadata such as project ID, friendly name, and creation timestamp. For detailed information about listing projects, pagination, and API response formats, see the Google BigQuery API documentation.

Configure Manually​

Google BigQuery API data sources can be manually configured to ingest data from any valid Google BigQuery API endpoint. Manual configuration provides maximum flexibility for accessing endpoints not covered by pre-built templates or when you need custom API configurations.

With manual configuration, you can also create more complex Google BigQuery API sources, such as sources that use custom query parameters, different HTTP methods, or custom data transformations.

Content Format​

  1. Select the format in which the data from the Google BigQuery API will be received from the Content Format pulldown menu. Nexla will automatically parse the data in the selected format. The Google BigQuery API typically returns data in JSON format.

API Endpoint URL​

  1. Enter the URL of the Google BigQuery API endpoint from which this source will fetch data in the Set API URL field. This should be the complete URL including the protocol (https://) and any required path parameters. Google BigQuery API endpoints typically follow the pattern https://bigquery.googleapis.com/bigquery/v2/{endpoint_path} or https://content.googleapis.com/bigquery/v2/{endpoint_path}.

Ensure the API endpoint URL is correct and accessible with your current credentials. You can test the endpoint using the Test button after configuring the URL. The endpoint requires OAuth2 authentication via the Authorization: Bearer {token} header, which is handled automatically by your credential configuration. The access token is obtained through the OAuth2 flow and is automatically refreshed when it expires. The application must have the appropriate permissions in your Google Cloud project to access the endpoint. For detailed information about Google BigQuery API endpoints, authentication, and available APIs, see the Google BigQuery API documentation.