Skip to main content

Data Source

The Gutendex API provides access to metadata for over 70,000 public domain books from Project Gutenberg, including titles, authors, languages, download counts, and available formats. Follow the instructions below to create a new data flow that ingests data from a Gutendex API source in Nexla.
gutendex_api.png

Gutendex API

Create a New Data Flow​

  1. To create a new data flow, navigate to the Integrate section, and click the New Data Flow button. Then, select the desired flow type from the list, and click the Create button.

  2. Select the Gutendex API connector tile from the list of available connectors. Then, select the credential that will be used to connect to the Gutendex API instance, and click Next; or, create a new Gutendex API credential for use in this flow.

  3. In Nexla, Gutendex API data sources can be created using pre-built endpoint templates, which expedite source setup for common Gutendex API endpoints. Each template is designed specifically for the corresponding Gutendex API endpoint, making data source setup easy and efficient.
    • To configure this source using a template, follow the instructions in Configure Using a Template.

    Gutendex API sources can also be configured manually, allowing you to ingest data from Gutendex API endpoints not included in the pre-built templates or apply further customizations to exactly suit your needs.
    • To configure this source manually, follow the instructions in Configure Manually.

Configure Using a Template​

Nexla provides pre-built templates that can be used to rapidly configure data sources to ingest data from common Gutendex API endpoints. Each template is designed specifically for the corresponding Gutendex API endpoint, making data source setup easy and efficient.

Endpoint Settings​

  • Select the endpoint from which this source will fetch data from the Endpoint pulldown menu. Available endpoint templates are listed in the expandable boxes below. Click on an endpoint to see more information about it and how to configure your data source for this endpoint.

    List books

    Retrieves a paginated list of books from Project Gutenberg, with support for filtering by sort order, language codes, search terms, and topic. Use this endpoint to bulk-ingest book metadata or to build filtered catalogs for specific subjects or languages.

    • Sends a GET request to /books using the base URL configured in your Gutendex API credential, with optional query parameters for Sort, Language Codes, Search, and Topic.
    • Response data is extracted from $.results[*], returning each book as an individual record. Each response page contains up to 32 records; Nexla automatically follows pagination links to retrieve the full result set.
    • Configure the following parameters: Sort — result ordering (e.g., popular, ascending, descending). Language Codes — comma-separated ISO 639-1 language codes to filter by (e.g., en, fr,fi). Search — space-separated keywords to search author names and titles. Topic — filter by bookshelf or subject phrase.

    The Gutendex dataset is a static catalog of public domain books — it does not update in real time. For most use cases, a one-time or infrequent scheduled ingestion is sufficient.

    Get book by ID

    Retrieves detailed metadata for a single book using its Project Gutenberg ID number. Use this endpoint when you need the full metadata record for a specific known book.

    • Sends a GET request to /books/{id} using the base URL configured in your Gutendex API credential, where {id} is the numeric Project Gutenberg book ID.
    • Response data is extracted from $, returning the full book metadata as a single record.
    • Configure the following parameters: Book ID — the numeric Project Gutenberg ID of the book to retrieve (e.g., 1342 for Pride and Prejudice).

    Project Gutenberg book IDs can be found in the URL of any book's page on gutenberg.org, or retrieved from the List books endpoint. Use a lookup macro to pass book IDs dynamically when chaining API calls.

Endpoint Testing​

Once the selected endpoint template has been configured, Nexla can retrieve a sample of the data that will be fetched according to the current settings. This allows users to verify that the source is configured correctly before saving.

  • To test the current endpoint configuration, click the Test button to the right of the endpoint selection menu. Sample data will be fetched & displayed in the Endpoint Test Result panel on the right.

  • If the sample data is not as expected, review the selected endpoint and associated settings, and make any necessary adjustments. Then, click the Test button again, and check the sample data to ensure that the correct information is displayed.

Configure Manually​

Gutendex API sources can be configured to ingest book metadata from the Gutendex /books endpoint. The endpoint supports a rich set of query parameters for filtering by language, topic, author, copyright status, and more, giving you precise control over which records are ingested. Configuration options allow Gutendex API sources to be fully customized to suit any use case.

First, select the API method that will be used for calls to the Gutendex API from the Method pulldown menu. For retrieving book metadata, select GET.

API Endpoint URL​

  1. Enter the URL of the Gutendex API endpoint from which this source will fetch data in the Set API URL field. The primary endpoint for listing and filtering books is:

    https://gutendex.com/books

    To retrieve a single book by its Project Gutenberg ID, use:

    https://gutendex.com/books/{id}

    where {id} is the numeric Project Gutenberg ID of the book (for example, https://gutendex.com/books/1342 retrieves the record for Pride and Prejudice).

If your organization runs a self-hosted Gutendex instance, replace https://gutendex.com with the base URL of your private server. The base URL is configured in the Gutendex API credential — ensure the credential's Base URL field matches the server your source will query.

Query Parameters​

Gutendex supports a set of query parameters that can be appended to the /books URL to filter and sort results. Append parameters to the endpoint URL in the Set API URL field using standard URL query string format (e.g., https://gutendex.com/books?languages=en&topic=fiction).

The following parameters are available:

  • author_year_start — Returns only books that have at least one author who was alive on or after the specified year. Enter an integer year value (e.g., author_year_start=1800).

  • author_year_end — Returns only books that have at least one author who was alive on or before the specified year. Enter an integer year value (e.g., author_year_end=1899). Use author_year_start and author_year_end together to find books by authors alive within a specific century or period (e.g., author_year_start=1800&author_year_end=1899 returns books by authors alive in the 19th century).

  • copyright — Filters books by copyright status. Accepted values are true (books with existing copyrights), false (books in the public domain in the USA), or null (books with no available copyright information).

  • ids — Returns only books whose Project Gutenberg ID numbers match the provided comma-separated list (e.g., ids=1342,11,1661). This is useful when ingesting a specific known set of books.

  • languages — Filters books to those available in the specified language(s). Enter a comma-separated list of two-character ISO 639-1 language codes (e.g., languages=en for English, languages=fr,fi for French or Finnish).

  • mime_type — Filters books to those that have at least one download format matching the specified MIME type prefix (e.g., mime_type=text/ matches text/html, text/plain, and similar types; mime_type=application/epub matches EPUB formats).

  • search — Searches author names and book titles for the specified space-separated keywords (e.g., search=dickens+great+expectations).

  • sort — Controls the order of results. Accepted values are popular (default, ordered by number of Project Gutenberg downloads), ascending (ordered by Project Gutenberg ID in ascending order), and descending (ordered by ID in descending order).

  • topic — Searches for books whose bookshelves or subjects contain the specified case-insensitive phrase (e.g., topic=children matches books on the "Children's Literature" bookshelf or books with subjects containing "children").

The Gutendex API returns paginated results with up to 32 book records per page. Each response includes a next field containing the URL for the next page of results. Nexla automatically follows pagination links to retrieve the full result set when ingesting data from this endpoint.

Date/Time Macros (API URL)​

Optional

Optionally, the API URL can be customized using macros—all macros added to the API URL will be converted into values when Nexla executes the API call. Macros are dynamic placeholders that allow you to create flexible API endpoints that can adapt to different time periods or data requirements.

Date/time macros are not typically required for Gutendex API sources, as the Gutendex dataset is a static catalog of public domain books rather than a time-series data source. However, macros can be useful if you are using a self-hosted Gutendex instance that has been extended with time-based filtering capabilities.

  1. To add a macro, type { at the appropriate position in the API URL (within the Set API URL field), and select the desired macro from the dropdown list.

    • `{now}` – The current datetime
    • `{now-1}` – The datetime one time unit before the current datetime
    • `{now+1}` – The datetime one time unit after the current datetime
    • custom – Datetime macros can reference any number of time units before or after the current datetime—for example, enter `(now-4)` to indicate the datetime four time units before the current datetime

  2. Select the format that will be applied to datetime macros from the Date Format for Date/Time Macro pulldown menu. This format will be applied to the base datetime value of the macro—i.e., the value of {now} in {now-1}.

  3. Select the datetime unit that will be used to perform mathematical operations in the included macro(s) from the Time Unit for Operations pulldown menu—for example, for the macro {now-1}, when Day is selected, {now-1} will be converted to the datetime one day before the current datetime.

Lookup-Based Macros (API URL)​

Optional

Column values from existing lookups can also be included as macros in the API URL. Lookup-based macros allow you to reference data from previously configured data sources or lookups, enabling dynamic API endpoints that can adapt based on existing data.

Lookup-based macros are particularly useful for the Gutendex ids parameter — for example, if you maintain a separate Nexla data source containing a list of Project Gutenberg book IDs of interest, you can use a lookup macro to dynamically pass those IDs to the Gutendex /books endpoint.

  1. To include a lookup column value macro, select the relevant lookup from the Add Lookups to Supported Macros pulldown menu.

  2. Type { at the appropriate position in the API URL, and select the lookup column-based macro from the dropdown list. Lookup-based macros are automatically populated into the macro list when a lookup is selected in the Add Lookups to Supported Macros pulldown menu.

Path to Data​

Optional

If only a subset of the data returned by the Gutendex API endpoint is needed, you can designate the part of the response that should be included in the Nexset(s) produced from this source by specifying the path to the relevant data within the response.

The Gutendex /books endpoint returns a paginated JSON response in the following structure:

{
  "count": 70000,
  "next": "https://gutendex.com/books?page=2",
  "previous": null,
  "results": [...]
}

The results array contains the book objects. To configure Nexla to treat each element of this array as a separate record, specify the JSON path to the results array.

Path to Data is recommended for Gutendex API sources. Without specifying $.results[*] as the path, Nexla will treat the entire paginated response object (including count, next, and previous) as a single record rather than ingesting individual book records.

  • To specify the path to book records in the Gutendex API response, enter $.results[*] in the Set Path to Data in Response field.

    Path to Data Example:

    For the Gutendex /books endpoint, the relevant data is in the top-level results array. Enter $.results[*] in the Set Path to Data in Response field to configure Nexla to ingest each book object as a separate record.

Autogenerate Path Suggestions​

Nexla can also autogenerate data path suggestions based on the response from the API endpoint. These suggested paths can be used as-is or modified to exactly suit your needs.

  • To use this feature, click the Test button next to the Set API URL field to fetch a sample response from the API endpoint. Suggested data paths generated based on the content & format of the response will be displayed in the Suggestions box below the Set Path to Data in Response field.

  • Click on a suggestion to automatically populate the Set Path to Data in Response field with the corresponding path. The populated path can be modified directly within the field if further customization is needed.

    PathSuggestions.png

Metadata​

If metadata is included in the response but is located outside of the defined path to relevant data, you can configure Nexla to include this data as common metadata in each record.

For the Gutendex /books endpoint, the top-level count, next, and previous fields sit outside the results array. If any of this contextual information (such as the total count of matching books) should be preserved alongside each individual book record, specify the path to that metadata.

Metadata paths are particularly useful for preserving pagination context from the Gutendex API response. For example, including the count field as metadata allows downstream processes to know the total number of books that matched the query.

  • To specify the location of metadata that should be included with each record, enter the path to the relevant metadata in the Path to Metadata in Response field.

    • For responses in JSON format, enter the JSON path to the object or array that contains the metadata. For example, to include the top-level count field from Gutendex responses, enter $.count.

Request Headers​

Optional

  • If Nexla should include any additional request headers in API calls to this source, enter the headers & corresponding values as comma-separated pairs in the Request Headers field (e.g., header1:value1,header2:value2). The Gutendex public API does not require any additional headers beyond standard HTTP defaults.

    You do not need to include any headers already present in the credentials. The Gutendex API does not require authentication headers — standard GET requests to the public endpoint work without any additional header configuration.

Endpoint Testing​

After configuring all settings for the selected endpoint, Nexla can retrieve a sample of the data that will be fetched according to the current configuration. This allows users to verify that the source is configured correctly before saving.

  • To test the current endpoint configuration, click the Test button to the right of the endpoint selection menu. Sample data will be fetched & displayed in the Endpoint Test Result panel on the right.

  • If the sample data is not as expected, review the selected endpoint and associated settings, and make any necessary adjustments. Then, click the Test button again, and check the sample data to ensure that the correct information is displayed.

Save & Activate the Source​

  1. Once all of the relevant steps in the above sections have been completed, click the Create button in the upper right corner of the screen to save and create the new Gutendex API data source. Nexla will now begin ingesting data from the configured endpoint and will organize any data that it finds into one or more Nexsets.