Skip to main content

Google BigQuery API Destination

Nexla's bi-directional connectors allow data to flow both to and from any location, making it simple to create a FlexFlow, DB-CDC, Spark ETL, or Replication data flow that sends data to a Google BigQuery API location.
gbq_api.png

Google BigQuery API

Create a Google BigQuery API Destination​

  1. Click the + icon on the Nexset that will be sent to the Google BigQuery API destination, and select the Send to Destination option from the menu.

  2. Select the Google BigQuery API connector from the list of available destination connectors. Then, select the credential that will be used to connect to the Google BigQuery API, and click Next; or, create a new Google BigQuery API credential for use in this flow.

  3. In Nexla, Google BigQuery API destinations can be created using pre-built endpoint templates, which expedite destination setup for common Google BigQuery API endpoints. Each template is designed specifically for the corresponding Google BigQuery API endpoint, making destination configuration easy and efficient.
    • To configure this destination using a template, follow the instructions in Configure Using a Template.

    Google BigQuery API destinations can also be configured manually, allowing you to send data to Google BigQuery API endpoints not included in the pre-built templates or apply further customizations to exactly suit your needs.
    • To configure this destination manually, follow the instructions in Configure Manually.

Configure Using a Template​

Nexla provides pre-built templates that can be used to rapidly configure destinations to send data to common Google BigQuery API endpoints. Each template is designed specifically for the corresponding Google BigQuery API endpoint, making destination setup easy and efficient.

  • To configure this destination using a template, select the endpoint to which data will be sent from the Endpoint pulldown menu. Then, click on the template in the list below to expand it, and follow the instructions to configure additional endpoint settings.

    Stream Data Into Table

    Streams records from a Nexset into BigQuery one record at a time without needing to run a load job.

    • Enter the BigQuery project ID in the Project ID field. This is the unique identifier for your Google Cloud project that contains the BigQuery dataset and table. The project ID is used to construct the API endpoint URL for streaming data into the specified table.
    • Enter the BigQuery dataset ID in the BigQuery Dataset Name field. This is the identifier for the BigQuery dataset that contains the table you want to stream data into. The dataset ID is used to construct the API endpoint URL.
    • Enter the BigQuery table ID in the Bigquery Table Name field. This is the identifier for the BigQuery table you want to stream data into. The table ID is used to construct the API endpoint URL.
    • The endpoint uses POST requests to https://bigquery.googleapis.com/bigquery/v2/projects/{projectId}/datasets/{datasetId}/tables/{tableId}/insertAll where {projectId}, {datasetId}, and {tableId} are the values you configured. The endpoint URL is automatically constructed based on the values you specify.
    • The endpoint sends data from your Nexset as the request body in JSON format. The request body is automatically formatted as {"rows": [{"json": {message.json}}]} where each record from your Nexset is wrapped in a rows array with a json object containing the record data. Each record is streamed individually into BigQuery without requiring a load job.
    • The endpoint uses automatic schema mapping (mapping.mode: auto) to map fields from your Nexset to BigQuery table columns. The mapping is determined automatically based on field names and data types. No tracker mode is used (tracker_mode: NONE), which means the destination does not track which records have been successfully inserted.
    • This endpoint does not automatically create a data source to track the responses received from the Google BigQuery API after each call.

    The request body must be properly formatted JSON that matches the Google BigQuery API specification for streaming inserts. The endpoint requires OAuth2 authentication via the Authorization: Bearer {token} header, which is handled automatically by your credential configuration. The Content-Type: application/json header is automatically included in requests. The endpoint streams data directly into BigQuery tables without requiring a load job, which makes it suitable for real-time data ingestion. The automatic schema mapping ensures that fields from your Nexset are correctly mapped to BigQuery table columns based on field names. The table must already exist in BigQuery before you can stream data into it. For detailed information about streaming inserts, including request formats, schema mapping, and API response formats, see the Google BigQuery API documentation.

Configure Manually​

Google BigQuery API destinations can be manually configured to send data to any valid Google BigQuery API endpoint. Manual configuration provides maximum flexibility for accessing endpoints not covered by pre-built templates or when you need custom API configurations.

With manual configuration, you can also create more complex Google BigQuery API destinations, such as destinations that use custom request headers, different HTTP methods, or custom data formatting.

Content Format​

  1. Select the format in which the Nexset data will be sent to the Google BigQuery API from the Content Format pulldown menu. Nexla will automatically convert the data to the selected format for each API call. The Google BigQuery API typically uses JSON format for request bodies.

API Endpoint URL​

  1. Enter the URL of the Google BigQuery API endpoint to which you want to send the Nexset data in the URL field. This should be the complete URL including the protocol (https://) and any required path parameters. Google BigQuery API endpoints typically follow the pattern https://bigquery.googleapis.com/bigquery/v2/{endpoint_path} or https://content.googleapis.com/bigquery/v2/{endpoint_path}.

Ensure the API endpoint URL is correct and accessible with your current credentials. You can test the endpoint using the Test button after configuring the URL. The endpoint requires OAuth2 authentication via the Authorization: Bearer {token} header, which is handled automatically by your credential configuration. The access token is obtained through the OAuth2 flow and is automatically refreshed when it expires. The application must have the appropriate permissions in your Google Cloud project to perform the actions you want to use. For detailed information about Google BigQuery API endpoints, authentication, and available APIs, see the Google BigQuery API documentation.