Hugging Face Datasets

Hugging Face Datasets connects to the Hugging Face Hub and Datasets Server REST APIs, letting you query dataset rows and splits, search models, datasets, Spaces, and papers, and read organization and account information. As a destination, it can commit changes to repositories, manage collections, webhooks, and SCIM users, and create or move repositories on the Hub.

Power end-to-end data operations for your Hugging Face Datasets API with Nexla. Our bi-directional Hugging Face Datasets connector is purpose-built for Hugging Face Datasets, making it simple to ingest data, sync it across systems, and deliver it anywhere — all with no coding required. Nexla turns API-sourced data into ready-to-use, reusable data products and makes it easy to send data to Hugging Face Datasets or any other destination. With comprehensive monitoring, lineage tracking, and access controls, Nexla keeps your Hugging Face Datasets workflows fast, secure, and fully governed.

Features

Type: API

SourceDestination

• Seamless API Integration: Connect to any endpoint as source or destination without coding, with automatic data product creation
• Visual Composition & Chaining: Build complex integrations using visual templates, chain API calls, and compose workflows with data validation and filtering
• API Proxy: Expose curated slices of your data securely with a secure and customizable API proxy that validates and transforms data on the fly
• Request optimization with intelligent batching, retry, and caching to minimize API calls and costs

Prerequisites

Public read access to the Hugging Face Datasets Server and Hub APIs does not require credentials, so you can query public datasets and splits with No Authentication. To access private repositories, read account or organization details, or perform any write operation, you must supply a Hugging Face User Access Token.

To create a User Access Token:

1. Sign in to your Hugging Face account.

2. Open your account Access Tokens settings page (Settings > Access Tokens).

3. Click New token (or Create new token).

4. Enter a descriptive name for the token (e.g., "Nexla Integration") and choose the role that matches your use case:

   • read — read access to repositories you can already read, including private repositories you or your organizations own. Use this for ingesting data only.
   • write — read access plus write access to repositories you can write to. Use this to commit changes, create repositories, or manage Hub resources as a destination.
   • fine-grained — scoped access to specific resources; recommended for production. Grant the specific permissions your flow needs.

5. Click Create token, then copy the token value immediately. The token is shown only once and begins with the hf_ prefix.

6. Store the token securely. You will paste it into your Nexla credential.

The token is sent as a bearer token in the Authorization header on API requests. If a token is leaked, revoke or refresh it from the Access Tokens settings page and generate a new one. For more detail, see the User Access Tokens documentation and the Datasets Server documentation.

Authenticate

Credentials required

No Authentication

Public API access to Hugging Face Datasets Server. No credentials required.

Field	Required	Secret	Description
Base URL	Yes	No	The base URL for the Hugging Face Hub API.
Datasets Server Base URL	Yes	No	The base URL for the Hugging Face Datasets Server API.

API Key Authentication

Authenticate using a Hugging Face User Access Token. Required for private repositories and write operations.

Field	Required	Secret	Description
API Key	Yes	Yes	Your Hugging Face User Access Token. Generate one at https://huggingface.co/settings/tokens. Example: hf_...
Base URL	Yes	No	The base URL for the Hugging Face Hub API.
Datasets Server Base URL	Yes	No	The base URL for the Hugging Face Datasets Server API.

Create a credential in Nexla

1. After selecting the data source/destination type, click the Add Credential tile to open the Add New Credential overlay.

2. Enter a name for the credential in the Credential Name field and a short, meaningful description in the Credential Description field.

3. Select the authentication method that matches your use case. Choose No Authentication to query public datasets, or API Key Authentication to supply a Hugging Face User Access Token for private repositories and write operations.

4. If using API Key Authentication, enter the User Access Token you obtained in Prerequisites in the API Key field. Confirm or adjust the Base URL and Datasets Server Base URL fields, which point to the Hugging Face Hub API and Datasets Server API respectively.

   The User Access Token is sent as a bearer token in the Authorization header on all API requests. Keep it confidential. If the token is compromised, revoke or refresh it from your Hugging Face Access Tokens settings and generate a new one.

5. Click the Save button at the bottom of the overlay. The newly added credential will now appear in a tile on the Authenticate screen during data source/destination creation.

Use as a data source

To create a new data flow, navigate to the Integrate section, and click the New Data Flow button. Select the Hugging Face Datasets connector tile, then select the credential that will be used to connect to Hugging Face, and click Next; or, create a new Hugging Face Datasets credential for use in this flow.

Endpoint templates

Nexla provides pre-built templates that can be used to rapidly configure data sources to ingest data from common Hugging Face endpoints. Select the endpoint from which this source will fetch data from the Endpoint pulldown menu. Available endpoint templates are listed in the expandable boxes below.

[Rest API] Get Dataset Rows

Retrieve rows from a Hugging Face dataset for a specific configuration and split.

For details about response structure, configurations, splits, and pagination, see the Hugging Face Datasets Server documentation.

[Rest API] Get Dataset Splits

Retrieve the available splits for a Hugging Face dataset.

For details about configurations and splits, see the Hugging Face Datasets Server documentation.

[Rest API] List user repositories with storage info

Returns a paginated list of the authenticated user's repositories with storage usage information.

This endpoint requires a valid User Access Token. Nexla will follow pagination to fetch subsequent pages of data.

[Rest API] Get organization members

List members of an organization with optional filtering and cursor-based pagination.

This endpoint requires a valid User Access Token with appropriate organization access. Nexla will follow cursor-based pagination to fetch subsequent pages of data.

[Rest API] List SCIM users

List SCIM users in an organization with pagination and optional filtering.

SCIM endpoints require an Enterprise organization and appropriate administrative permissions on your User Access Token.

[Rest API] List SCIM groups

Returns a paginated list of SCIM groups in the organization.

SCIM endpoints require an Enterprise organization and appropriate administrative permissions on your User Access Token.

[Rest API] Get Current User Info

Returns information about the currently authenticated user.

This endpoint requires a valid User Access Token.

[Rest API] Quick search

Performs a quick search across models, datasets, spaces, and discussions on Hugging Face Hub. Note: omit unused query parameters or pass empty strings; the API ignores empty-valued params.

For details about search parameters and response structure, see the Hugging Face Hub API documentation.

[Rest API] Search Papers

Searches the Hugging Face papers database for papers matching the provided query string.

For details about the papers database and response structure, see the Hugging Face Hub API documentation.

[Rest API] List collections

Retrieve a paginated list of collections with optional filtering and sorting.

Nexla will follow pagination to fetch subsequent pages of data. For details, see the Hugging Face Hub API documentation.

[Rest API] List models on the Hugging Face Hub

List models on the Hugging Face Hub with filter, search, author, and sort parameters.

For details about filter, search, author, and sort parameters, see the Hugging Face Hub API documentation.

[Rest API] List Spaces on the Hugging Face Hub

List Spaces on the Hugging Face Hub with filter, search, and sort parameters.

For details about filter, search, and sort parameters, see the Hugging Face Hub API documentation.

Once the selected endpoint template has been configured, click the Test button to the right of the endpoint selection menu to retrieve a sample of the data that will be fetched. Sample data will be displayed in the Endpoint Test Result panel on the right, allowing you to verify that the source is configured correctly before saving.

Manual configuration

Hugging Face Datasets data sources can also be manually configured to ingest data from any valid Hugging Face Hub or Datasets Server API endpoint, including endpoints not covered by the pre-built templates, chained API calls, or custom request parameters. Select the Advanced tab at the top of the configuration screen, and follow the instructions in Connect to Any API to configure the API method, endpoint URL, date/time and lookup macros, path to data, metadata, and request headers.

Once all of the relevant settings have been configured, click the Create button in the upper right corner of the screen to save and create the new Hugging Face Datasets data source. Nexla will now begin ingesting data from the configured endpoint and will organize any data that it finds into one or more Nexsets.

Use as a destination

Click the + icon on the Nexset that will be sent to the Hugging Face destination, and select the Send to Destination option from the menu. Select the Hugging Face Datasets connector from the list of available destination connectors, then select the credential that will be used to connect to Hugging Face, and click Next; or, create a new Hugging Face Datasets credential for use in this flow.

Endpoint templates

Nexla provides pre-built templates that can be used to rapidly configure destinations to send data to common Hugging Face endpoints. Select the endpoint to which data will be sent from the Endpoint pulldown menu. Then, click on the template in the list below to expand it, and follow the instructions to configure additional endpoint settings.

[Rest API] Create webhook

Creates a new webhook for monitoring repository events and triggering notifications.

Each record from your Nexset is sent as a separate request. The request body must match the format expected by the Hugging Face API. See the Hugging Face Hub API documentation.

[Rest API] Update webhook

Updates the configuration of an existing webhook.

Each record from your Nexset is sent as a separate request. The request body must match the format expected by the Hugging Face API. See the Hugging Face Hub API documentation.

[Rest API] Change member role

Update the role and resource group permissions of an organization member.

This endpoint requires appropriate organization administrative permissions on your User Access Token.

[Rest API] Create a SCIM user

Creates a new SCIM user in the organization.

SCIM endpoints require an Enterprise organization and appropriate administrative permissions on your User Access Token.

[Rest API] Update a SCIM user

Updates a SCIM user with complete replacement of user attributes.

SCIM endpoints require an Enterprise organization and appropriate administrative permissions on your User Access Token.

[Rest API] Partially update a SCIM user

Partially updates a SCIM user using JSON Patch operations.

SCIM endpoints require an Enterprise organization and appropriate administrative permissions on your User Access Token.

[Rest API] Register OAuth Application

Register a new OAuth application with credentials and configuration.

Each record from your Nexset is sent as a separate request. The request body must match the format expected by the Hugging Face API. See the Hugging Face Hub API documentation.

[Rest API] Commit model changes

Creates or updates a commit on a specific model repository revision with optional pull request creation.

This endpoint requires a User Access Token with write access to the target repository. See the Hugging Face Hub API documentation.

[Rest API] Commit dataset changes

Creates or updates a commit on a specific dataset repository revision with optional pull request creation.

This endpoint requires a User Access Token with write access to the target repository. See the Hugging Face Hub API documentation.

[Rest API] Commit space changes

Creates or updates a commit on a specific space repository revision with optional pull request creation.

This endpoint requires a User Access Token with write access to the target repository. See the Hugging Face Hub API documentation.

[Rest API] Create a new repository

Creates a new repository with the specified configuration. Requires API key authentication.

This endpoint requires a User Access Token with write access. See the Hugging Face Hub API documentation.

[Rest API] Create a collection

Create a new collection with title, description, and optional items.

Each record from your Nexset is sent as a separate request. The request body must match the format expected by the Hugging Face API. See the Hugging Face Hub API documentation.

[Rest API] Update collection metadata

Update metadata (title, description, theme) of an existing collection.

Each record from your Nexset is sent as a separate request. The request body must match the format expected by the Hugging Face API. See the Hugging Face Hub API documentation.

[Rest API] Delete a collection

Delete a collection from the Hub.

This operation is destructive and cannot be undone. Ensure your token has the required permissions before activating this destination.

[Rest API] Move or rename a repository

Move (rename) a repository to a new namespace or repository name.

This endpoint requires a User Access Token with write access to the repository. See the Hugging Face Hub API documentation.

[Rest API] Delete a model, dataset, or space repository

Delete a model, dataset, or space repository.

This operation is destructive and cannot be undone. Ensure your token has the required permissions before activating this destination.

[Rest API] Delete a webhook registration

Delete a webhook registration.

This operation is destructive and cannot be undone. Ensure your token has the required permissions before activating this destination.

[Rest API] Remove a SCIM-provisioned user from an organization

Remove a SCIM-provisioned user from an organization.

SCIM endpoints require an Enterprise organization and appropriate administrative permissions on your User Access Token. This operation is destructive.

Manual configuration

Hugging Face Datasets destinations can also be manually configured to send data to any valid Hugging Face Hub or Datasets Server API endpoint. Select the Advanced tab at the top of the configuration screen, and follow the instructions in Connect to Any API to configure the API method, data format, endpoint URL, request headers, attribute exclusions, record batching, and response webhooks.

Hugging Face APIs typically expect JSON format for most operations. For update operations, include the identifier of the object to be updated in the request URL.

Save & activate

Once all endpoint settings have been configured, click the Done button in the upper right corner of the screen to save and create the destination. To send the data to the configured Hugging Face endpoint, open the destination resource menu, and select Activate.

The Nexset data will not be sent to the Hugging Face endpoint until the destination is activated. Destinations can be activated immediately or at a later time, providing full control over data movement.