Google BigQuery API

Google BigQuery is a fully managed, serverless data warehouse that enables organizations to analyze massive datasets using SQL queries, providing real-time analytics, machine learning integration, and seamless data processing capabilities to support business intelligence and data-driven decision making.

Power end-to-end data operations for your Google BigQuery API API with Nexla. Our bi-directional Google BigQuery API connector is purpose-built for Google BigQuery API, making it simple to ingest data, sync it across systems, and deliver it anywhere — all with no coding required. Nexla turns API-sourced data into ready-to-use, reusable data products and makes it easy to send data to Google BigQuery API or any other destination. With comprehensive monitoring, lineage tracking, and access controls, Nexla keeps your Google BigQuery API workflows fast, secure, and fully governed.

Features

Type: API

SourceDestination

• Seamless API Integration: Connect to any endpoint as source or destination without coding, with automatic data product creation
• Visual Composition & Chaining: Build complex integrations using visual templates, chain API calls, and compose workflows with data validation and filtering
• API Proxy: Expose curated slices of your data securely with a secure and customizable API proxy that validates and transforms data on the fly
• Request optimization with intelligent batching, retry, and caching to minimize API calls and costs

Prerequisites

Before creating a Google BigQuery API credential, you need to have a Google Cloud Platform (GCP) account with BigQuery API enabled and create OAuth2 credentials (Client ID and Client Secret) in the Google Cloud Console. The OAuth2 credentials are required to authenticate with the Google BigQuery API.

To obtain your OAuth2 credentials, follow these steps:

1. Sign in to the Google Cloud Console with your Google account.

2. Create a new project or select an existing project:

   • Click the project dropdown at the top of the page.
   • Click New Project to create a new project, or select an existing project from the list.
   • Provide a project name and click Create.

3. Enable the BigQuery API for your project:

   • Navigate to APIs & Services > Library in the Google Cloud Console.
   • Search for "BigQuery API" and click on it.
   • Click Enable to enable the BigQuery API for your project.

4. Configure the OAuth2 consent screen:

   • Navigate to APIs & Services > OAuth consent screen.
   • Select External (unless you're using a Google Workspace account, in which case you can select Internal).
   • Fill in the required application information (app name, user support email, developer contact information).
   • Add the required scopes. For BigQuery API access, add the scope https://www.googleapis.com/auth/bigquery.
   • Click Save and Continue through the remaining steps.

5. Create OAuth2 credentials:

   • Navigate to APIs & Services > Credentials.
   • Click Create Credentials > OAuth client ID.
   • Select Web application as the application type.
   • Provide a name for your OAuth client.
   • Add authorized redirect URIs if required (Nexla will handle the OAuth flow).
   • Click Create to generate your OAuth2 credentials.
   • Copy the Client ID and Client Secret immediately, as the Client Secret may not be displayed again.

6. Store the Client ID and Client Secret securely, as you will need them to configure your Nexla credential.

The OAuth2 credentials are used to authenticate API requests to the BigQuery API. The default scope for BigQuery API access is https://www.googleapis.com/auth/bigquery, which provides full access to BigQuery datasets and tables. You can use more restrictive scopes if you only need read-only access or specific permissions. The OAuth2 flow will redirect you to Google's authorization page where you'll sign in and grant permissions. For detailed information about creating OAuth2 credentials, setting up BigQuery API access, and managing permissions, refer to the Google Cloud documentation, Google OAuth2 documentation, and Google Cloud IAM documentation.

Authenticate

Credentials required

Field	Required	Secret	Description
GBQ Access Scope	No	No	GBQ Access Scope Allowed values: https://www.googleapis.com/auth/bigquery

Create a credential in Nexla

1. After selecting the data source/destination type, click the Add Credential tile to open the Add New Credential overlay.

New Credential Overlay – Google BigQuery API

2. Enter a name for the credential in the Credential Name field and a short, meaningful description in the Credential Description field.

   Google BigQuery API uses OAuth2 authentication for all API requests. The credential uses the standard Google OAuth2 flow to obtain access tokens that are used to authenticate each API call.

3. Enter the OAuth2 access scope in the GBQ Access Scope field. The default value is https://www.googleapis.com/auth/bigquery if not specified. This scope grants access to read and write data in BigQuery datasets and tables. The scope determines what permissions your application will have when accessing BigQuery resources.

   The OAuth2 access scope determines what permissions your application will have when accessing BigQuery resources. The default scope https://www.googleapis.com/auth/bigquery provides full access to BigQuery datasets and tables, including the ability to read and write data, create and delete datasets and tables, and manage jobs. You can use more restrictive scopes if you only need read-only access or specific permissions. For detailed information about available scopes and permissions, see the Google BigQuery API documentation and Google OAuth2 scopes documentation.

4. Click the Authorize button to begin the OAuth2 authorization flow. This will redirect you to Google's authorization page where you will be asked to sign in with your Google account and grant permissions to access your BigQuery resources.

   During the OAuth2 authorization flow, you will be redirected to Google's authorization page. You must sign in with a Google account that has access to the BigQuery projects and datasets you want to use. After signing in, you will be asked to grant permissions to access BigQuery resources based on the scope you configured. The permissions you grant determine what actions the application can perform with your BigQuery data. Once you grant permissions, you will be redirected back to Nexla, and the authorization will be completed automatically.

5. After completing the OAuth2 authorization flow, Nexla will automatically obtain and store the access token and refresh token needed to authenticate API requests. The access token is used to authenticate each API request, and the refresh token is used to obtain new access tokens when the current access token expires.

   OAuth2 access tokens expire after a certain period for security reasons. Nexla automatically refreshes expired access tokens using the refresh token, so you typically do not need to re-authorize unless you revoke access or the refresh token expires. The access token is sent in the Authorization header with the Bearer prefix for all API requests to the Google BigQuery API. The application must have the appropriate permissions in your Google Cloud project to perform the actions you want to use. For detailed information about OAuth2 authentication, token management, and BigQuery API permissions, see the Google Cloud documentation and Google OAuth2 documentation.

6. Click the Save button at the bottom of the overlay to save the configured credential. The newly added credential will now appear in a tile on the Authenticate screen during data source/destination creation and can be selected for use with a new data source or destination.

Use as a data source

To create a new data flow, navigate to the Integrate section, and click the New Data Flow button. Select the Google BigQuery API connector tile, then select the credential that will be used to connect to the Google BigQuery API, and click Next; or, create a new Google BigQuery API credential for use in this flow.

Endpoint templates

Nexla provides pre-built templates that can be used to rapidly configure data sources to ingest data from common Google BigQuery API endpoints. Select the endpoint from which this source will fetch data from the Endpoint pulldown menu. Available endpoint templates are listed in the expandable box below.

List all Projects

List projects to which the user has been granted any project role.

• This endpoint automatically lists all Google Cloud projects to which your authenticated account has been granted any project role. No additional configuration is required beyond selecting this endpoint template.
• The endpoint uses GET requests to https://bigquery.googleapis.com/bigquery/v2/projects to retrieve a list of all projects accessible to your account. The endpoint URL is automatically constructed based on the Google BigQuery API base URL.
• The endpoint uses pagination (iteration.type: paging.next.token) to retrieve all projects. Pagination uses the nextPageToken from the API response to navigate through pages. The pageToken query parameter is used to request the next page of results. The endpoint will automatically fetch all pages until all projects are retrieved.
• The response data path is $.projects[*], which extracts all project objects from the projects array in the API response. The next page token is extracted from $.nextPageToken for pagination.
• The response format is JSON, which is automatically parsed by Nexla.

This endpoint retrieves a list of all Google Cloud projects to which your authenticated account has been granted any project role. The endpoint requires OAuth2 authentication via the Authorization: Bearer {token} header, which is handled automatically by your credential configuration. The endpoint uses token-based pagination to retrieve all projects across multiple pages. The projects returned include project metadata such as project ID, friendly name, and creation timestamp. For detailed information about listing projects, pagination, and API response formats, see the Google BigQuery API documentation.

Once the selected endpoint template has been configured, click the Test button to the right of the endpoint selection menu to retrieve a sample of the data that will be fetched. Sample data will be displayed in the Endpoint Test Result panel on the right, allowing you to verify that the source is configured correctly before saving.

Manual configuration

Google BigQuery API data sources can also be manually configured to ingest data from any valid Google BigQuery API endpoint not covered by the pre-built templates. Select the Advanced tab at the top of the configuration screen, and follow the instructions in Connect to Any API to configure the API method, endpoint URL, date/time and lookup macros, path to data, metadata, and request headers.

Google BigQuery API endpoints typically follow the pattern https://bigquery.googleapis.com/bigquery/v2/{endpoint_path} or https://content.googleapis.com/bigquery/v2/{endpoint_path}. The endpoint requires OAuth2 authentication via the Authorization: Bearer {token} header, which is handled automatically by your credential configuration.

Once all of the relevant settings have been configured, click the Create button in the upper right corner of the screen to save and create the new Google BigQuery API data source. Nexla will now begin ingesting data from the configured endpoint and will organize any data that it finds into one or more Nexsets.

Use as a destination

Click the + icon on the Nexset that will be sent to the Google BigQuery API destination, and select the Send to Destination option from the menu. Select the Google BigQuery API connector from the list of available destination connectors, then select the credential that will be used to connect to the Google BigQuery API, and click Next; or, create a new Google BigQuery API credential for use in this flow.

Endpoint templates

Nexla provides pre-built templates that can be used to rapidly configure destinations to send data to common Google BigQuery API endpoints. Select the endpoint to which data will be sent from the Endpoint pulldown menu. Then, click on the template in the list below to expand it, and follow the instructions to configure additional endpoint settings.

Stream Data Into Table

Streams records from a Nexset into BigQuery one record at a time without needing to run a load job.

• Enter the BigQuery project ID in the Project ID field. This is the unique identifier for your Google Cloud project that contains the BigQuery dataset and table. The project ID is used to construct the API endpoint URL for streaming data into the specified table.
• Enter the BigQuery dataset ID in the BigQuery Dataset Name field. This is the identifier for the BigQuery dataset that contains the table you want to stream data into. The dataset ID is used to construct the API endpoint URL.
• Enter the BigQuery table ID in the Bigquery Table Name field. This is the identifier for the BigQuery table you want to stream data into. The table ID is used to construct the API endpoint URL.
• The endpoint uses POST requests to https://bigquery.googleapis.com/bigquery/v2/projects/{projectId}/datasets/{datasetId}/tables/{tableId}/insertAll where {projectId}, {datasetId}, and {tableId} are the values you configured. The endpoint URL is automatically constructed based on the values you specify.
• The endpoint sends data from your Nexset as the request body in JSON format. The request body is automatically formatted as {"rows": [{"json": {message.json}}]} where each record from your Nexset is wrapped in a rows array with a json object containing the record data. Each record is streamed individually into BigQuery without requiring a load job.
• The endpoint uses automatic schema mapping (mapping.mode: auto) to map fields from your Nexset to BigQuery table columns. The mapping is determined automatically based on field names and data types. No tracker mode is used (tracker_mode: NONE), which means the destination does not track which records have been successfully inserted.
• This endpoint does not automatically create a data source to track the responses received from the Google BigQuery API after each call.

The request body must be properly formatted JSON that matches the Google BigQuery API specification for streaming inserts. The endpoint requires OAuth2 authentication via the Authorization: Bearer {token} header, which is handled automatically by your credential configuration. The Content-Type: application/json header is automatically included in requests. The endpoint streams data directly into BigQuery tables without requiring a load job, which makes it suitable for real-time data ingestion. The automatic schema mapping ensures that fields from your Nexset are correctly mapped to BigQuery table columns based on field names. The table must already exist in BigQuery before you can stream data into it. For detailed information about streaming inserts, including request formats, schema mapping, and API response formats, see the Google BigQuery API documentation.

Manual configuration

Google BigQuery API destinations can also be manually configured to send data to any valid Google BigQuery API endpoint. Select the Advanced tab at the top of the configuration screen, and follow the instructions in Connect to Any API to configure the API method, data format, endpoint URL, request headers, attribute exclusions, record batching, and response webhooks.

Google BigQuery API endpoints typically follow the pattern https://bigquery.googleapis.com/bigquery/v2/{endpoint_path} or https://content.googleapis.com/bigquery/v2/{endpoint_path}. The endpoint requires OAuth2 authentication via the Authorization: Bearer {token} header, which is handled automatically by your credential configuration.

Save & activate

Once all endpoint settings have been configured, click the Done button in the upper right corner of the screen to save and create the destination. To send the data to the configured Google BigQuery API endpoint, open the destination resource menu, and select Activate.

The Nexset data will not be sent to the Google BigQuery API endpoint until the destination is activated. Destinations can be activated immediately or at a later time, providing full control over data movement.