Skip to main content

Jina AI Reader Data Source

The Jina AI Reader connector enables you to extract and summarize content from web pages, perform neural web searches, and retrieve structured information from URLs. This connector is particularly useful for applications that need to extract content from web pages, perform intelligent web searches, analyze web content, or build content aggregation systems. Follow the instructions below to create a new data flow that ingests data from a Jina AI Reader source in Nexla.
jina_ai_reader_api.png

Jina AI Reader

Create a New Data Flow​

  1. To create a new data flow, navigate to the Integrate section, and click the New Data Flow button. Then, select the desired flow type from the list, and click the Create button.

  2. Select the Jina AI Reader connector tile from the list of available connectors. Then, select the credential that will be used to connect to the Jina AI Reader API, and click Next; or, create a new Jina AI Reader credential for use in this flow.

  3. In Nexla, Jina AI Reader data sources can be created using pre-built endpoint templates, which expedite source setup for common Jina AI Reader API endpoints. Each template is designed specifically for the corresponding Jina AI Reader API endpoint, making source configuration easy and efficient.
    • To configure this source using a template, follow the instructions in Configure Using a Template.

    Jina AI Reader sources can also be configured manually, allowing you to ingest data from Jina AI Reader API endpoints not included in the pre-built templates or apply further customizations to exactly suit your needs.
    • To configure this source manually, follow the instructions in Configure Manually.

Configure Using a Template​

Nexla provides pre-built templates that can be used to rapidly configure data sources to ingest data from common Jina AI Reader API endpoints. Each template is designed specifically for the corresponding Jina AI Reader API endpoint, making data source setup easy and efficient.

Endpoint Settings​

  • Select the endpoint from which this source will fetch data from the Endpoint pulldown menu. Available endpoint templates are listed in the expandable boxes below. Click on an endpoint to see more information about it and how to configure your data source for this endpoint.

    Read and Summarize URL

    This endpoint fetches and summarizes a web page based on a provided URL using Jina AI Reader. Use this endpoint when you need to extract and summarize content from web pages, analyze web content, or retrieve structured information from URLs.

    • Enter the URL to fetch and process in the URL Encoded Input field. The URL must be URL-encoded (e.g., https%3A%2F%2Fexample.com%2Fpage for https://example.com/page). You can use standard URL encoding where special characters are replaced with percent-encoded values (e.g., : becomes %3A, / becomes %2F).

    The Read and Summarize URL endpoint uses GET requests to fetch web page content and automatically summarizes it using Jina AI's neural search technology. The endpoint returns both the summarized content and metadata about the page. For more information about the Read and Summarize URL endpoint, refer to the Jina AI Reader Documentation.

    Web Search

    This endpoint searches the web and returns results using neural search by Jina AI. Use this endpoint when you need to perform intelligent web searches, find relevant web content, or discover web pages based on search queries.

    • Enter the search query string in the URL Encoded Query field. The query must be URL-encoded (e.g., machine%20learning for machine learning). You can use standard URL encoding where spaces are replaced with %20 and special characters are percent-encoded.

    The Web Search endpoint uses GET requests to perform neural web searches and returns relevant search results with summaries. The endpoint leverages Jina AI's advanced neural search technology to provide intelligent search results. For more information about the Web Search endpoint, refer to the Jina AI Reader Documentation.

Endpoint Testing​

Once the selected endpoint template has been configured, Nexla can retrieve a sample of the data that will be fetched according to the current settings. This allows users to verify that the source is configured correctly before saving.

  • To test the current endpoint configuration, click the Test button to the right of the endpoint selection menu. Sample data will be fetched & displayed in the Endpoint Test Result panel on the right.

  • If the sample data is not as expected, review the selected endpoint and associated settings, and make any necessary adjustments. Then, click the Test button again, and check the sample data to ensure that the correct information is displayed.

Configure Manually​

Jina AI Reader data sources can be manually configured to ingest data from any valid Jina AI Reader API endpoint. Manual configuration provides maximum flexibility for accessing endpoints not covered by pre-built templates or when you need custom API configurations.

With manual configuration, you can also create more complex Jina AI Reader sources, such as sources that use chained API calls to fetch data from multiple endpoints or sources that require custom authentication headers or request parameters.

API Method​

  1. To manually configure this source, select the Advanced tab at the top of the configuration screen.

  2. Select the API method that will be used for calls to the Jina AI Reader API from the Method pulldown menu. The most common methods are:

    • GET: For retrieving data from the API (most Jina AI Reader endpoints use GET)

API Endpoint URL​

  1. Enter the URL of the Jina AI Reader API endpoint from which this source will fetch data in the Set API URL field. This should be the complete URL including the protocol (https://) and any required path parameters. Jina AI Reader API endpoints typically follow the pattern https://r.jina.ai/{url-encoded-url} for reading URLs or https://s.jina.ai/?q={url-encoded-query} for web searches.

Ensure the API endpoint URL is correct and accessible with your current credentials. You can test the endpoint using the Test button after configuring the URL. URLs and query parameters must be properly URL-encoded in the endpoint URL.

Path to Data​

Optional

If only a subset of the data that will be returned by API endpoint is needed, you can designate the part(s) of the response that should be included in the Nexset(s) produced from this source by specifying the path to the relevant data within the response. This is particularly useful when API responses contain metadata, pagination information, or other data that you don't need for your analysis.

For example, when a request call is used to fetch URL summaries, the API will typically return content data along with metadata. By entering the path to the relevant data, you can configure Nexla to extract the specific content you need.

Path to Data is essential when API responses have nested structures. Without specifying the correct path, Nexla might not be able to properly parse and organize your data into usable records. For Jina AI Reader API responses, common paths include $.data for content data or $.data[*] for arrays of results.

  • To specify which data should be treated as relevant in responses from this source, enter the path to the relevant data in the Set Path to Data in Response field.

    • For responses in JSON format enter the JSON path that points to the object or array that should be treated as relevant data. JSON paths use dot notation (e.g., $.data to access the data object).
    Path to Data Example:

    If the API response is in JSON format and includes a data object that contains the relevant content, the path to the response would be entered as $.data.

Autogenerate Path Suggestions​

Nexla can also autogenerate data path suggestions based on the response from the API endpoint. These suggested paths can be used as-is or modified to exactly suit your needs.

  • To use this feature, click the Test button next to the Set API URL field to fetch a sample response from the API endpoint. Suggested data paths generated based on the content & format of the response will be displayed in the Suggestions box below the Set Path to Data in Response field.

  • Click on a suggestion to automatically populate the Set Path to Data in Response field with the corresponding path. The populated path can be modified directly within the field if further customization is needed.

Metadata​

If metadata is included in the response but is located outside of the defined path to relevant data, you can configure Nexla to include this data as common metadata in each record. This is useful when you want to preserve important contextual information that applies to all records but isn't part of the main data array.

For example, when a request call is used to fetch URL summaries, the API response will typically include content data along with metadata such as page information, timestamps, or request IDs. In this case, if you have specified the path to the relevant data but metadata of interest is located in a different part of the response, you can specify a path to this metadata to include it with each record in the generated Nexset(s).

Metadata paths are particularly useful for preserving API response context like request IDs, timestamps, or page metadata that apply to all records in the response.

  • To specify the location of metadata that should be included with each record, enter the path to the relevant metadata in the Path to Metadata in Response field.

    • For responses in JSON format, enter the JSON path to the object or array that contains the metadata (e.g., $.meta for metadata objects).

Request Headers​

Optional

  • If Nexla should include any additional request headers in API calls to this source, enter the headers & corresponding values as comma-separated pairs in the Request Headers field (e.g., header1:value1,header2:value2). Additional headers are often required for API versioning, content type specifications, or custom authentication requirements.

    You do not need to include any headers already present in the credentials. Common headers like Authorization, Content-Type, and Accept are typically handled automatically by Nexla based on your credential configuration. For Jina AI Reader, you may want to include Accept:application/json to specify the response format.

Endpoint Testing​

After configuring all settings for the selected endpoint, Nexla can retrieve a sample of the data that will be fetched according to the current configuration. This allows users to verify that the source is configured correctly before saving.

  • To test the current endpoint configuration, click the Test button to the right of the endpoint selection menu. Sample data will be fetched & displayed in the Endpoint Test Result panel on the right.

  • If the sample data is not as expected, review the selected endpoint and associated settings, and make any necessary adjustments. Then, click the Test button again, and check the sample data to ensure that the correct information is displayed.

Save & Activate the Source​

  1. Once all of the relevant steps in the above sections have been completed, click the Create button in the upper right corner of the screen to save and create the new Jina AI Reader data source. Nexla will now begin ingesting data from the configured endpoint and will organize any data that it finds into one or more Nexsets.