Skip to main content

Data Source

Follow the instructions below to create a new data flow that ingests data from an OpenAI source in Nexla.
openai_api.png

Open AI

Create a New Data Flow​

  1. To create a new data flow, navigate to the Integrate section, and click the New Data Flow button. Then, select the desired flow type from the list, and click the Create button.

  2. Select the OpenAI connector tile from the list of available connectors. Then, select the credential that will be used to connect to your OpenAI account, and click Next; or, create a new OpenAI credential for use in this flow.

  3. In Nexla, OpenAI data sources can be created using pre-built endpoint templates, which expedite source setup for common OpenAI endpoints. Each template is designed specifically for the corresponding OpenAI endpoint, making source configuration easy and efficient.
    • To configure this source using a template, follow the instructions in Configure Using a Template.

    OpenAI sources can also be configured manually, allowing you to interact with OpenAI endpoints not included in the pre-built templates or apply further customizations to exactly suit your needs.
    • To configure this source manually, follow the instructions in Configure Manually.

Configure Using a Template​

Nexla provides pre-built templates that can be used to rapidly configure data sources to interact with common OpenAI endpoints. Each template is designed specifically for the corresponding OpenAI endpoint, making data source setup easy and efficient.

Endpoint Settings​

  • Select the endpoint from which this source will fetch data from the Endpoint pulldown menu. Available endpoint templates are listed in the expandable boxes below. Click on an endpoint to see more information about it and how to configure your data source for this endpoint.

    List Files

    This endpoint template returns a list of files that belong to your OpenAI organization. Use this template when you need to retrieve information about files you've uploaded to OpenAI, such as training data files, fine-tuning files, or other files used with OpenAI services.

    • This endpoint automatically retrieves all files from your OpenAI organization. No additional configuration is required beyond selecting this endpoint template.

    This endpoint returns a list of files that belong to your OpenAI organization, including file IDs, file names, file sizes, creation dates, and file purposes. Use this endpoint to discover which files are available in your OpenAI account before using them with other OpenAI endpoints.

    For detailed information about file management, API response structures, and available file data, see the OpenAI API documentation.

    List Assistants

    This endpoint template returns a list of assistants from your OpenAI account. Use this template when you need to retrieve information about your OpenAI assistants, including assistant configurations, models, and settings.

    • This endpoint automatically retrieves all assistants from your OpenAI account. The endpoint uses cursor-based pagination to handle large datasets efficiently.

    This endpoint returns a list of assistants from your OpenAI account, including assistant IDs, names, models, instructions, and configurations. The endpoint uses cursor-based pagination with after parameter to handle large datasets efficiently. Nexla will automatically fetch subsequent pages of data by using the cursor returned in the API response.

    For detailed information about assistant management, API response structures, pagination, and available assistant data, see the OpenAI API documentation.

    Chat Completions

    This endpoint template sends a chat completion request to the OpenAI API and retrieves the AI-generated response. Use this template when you need to interact with OpenAI's language models to generate text, answer questions, or perform natural language processing tasks.

    • Enter the model name in the Model field. This should be the name of the OpenAI model you want to use (e.g., gpt-4, gpt-3.5-turbo, gpt-4-turbo). The model name determines which OpenAI model will be used for processing requests. Refer to the OpenAI API documentation for available models.
    • Enter your message or prompt in the Message field. This should be the text you want to send to the OpenAI model for processing. The message is sent as part of a chat completion request with the role set to "user" and the content set to your message. The OpenAI model will process this message and generate an appropriate response.
    • Enter the temperature value in the Temperature field. This should be a number between 0 and 2 that controls the randomness of the model's output. Lower values make the output more deterministic, while higher values make it more creative. Typical values range from 0.7 to 1.0.
    • Enter the maximum number of tokens in the Max Tokens field. This should be the maximum number of tokens the model can generate in the response. The token limit helps control response length and API costs.

    This endpoint sends a POST request to the OpenAI chat completion endpoint with your message and model configuration in the request body. The request body format includes the model name, messages array, temperature, and max_tokens. The endpoint returns the AI-generated response from the OpenAI model.

    Different models may have different capabilities, response times, and costs. Temperature controls the randomness of the output, and max_tokens controls the response length. For detailed information about available models, chat completion formats, API endpoints, and response structures, see the OpenAI API documentation.

Endpoint Testing​

Once the selected endpoint template has been configured, Nexla can retrieve a sample of the data that will be fetched according to the current settings. This allows users to verify that the source is configured correctly before saving.

  • To test the current endpoint configuration, click the Test button to the right of the endpoint selection menu. Sample data will be fetched & displayed in the Endpoint Test Result panel on the right.

  • If the sample data is not as expected, review the selected endpoint and associated settings, and make any necessary adjustments. Then, click the Test button again, and check the sample data to ensure that the correct information is displayed.

Configure Manually​

OpenAI data sources can be manually configured to interact with any valid OpenAI API endpoint. Manual configuration provides maximum flexibility for accessing endpoints not covered by pre-built templates or when you need custom API configurations.

With manual configuration, you can also create more complex OpenAI sources, such as sources that use multiple API calls or sources that require custom request parameters or headers.

API Method​

  1. To manually configure this source, select the Advanced tab at the top of the configuration screen.

  2. Select the API method that will be used for calls to the OpenAI API from the Method pulldown menu. OpenAI API typically uses GET method for retrieving data and POST method for chat completions and other operations.

API Endpoint URL​

  1. Enter the URL of the OpenAI API endpoint from which this source will fetch data in the Set API URL field. This should be the complete URL to your OpenAI endpoint (e.g., https://api.openai.com/v1/files, https://api.openai.com/v1/assistants). Include any required query parameters in the URL.

Ensure the API endpoint URL is correct and accessible with your current credentials. The OpenAI API endpoint URL should point to the specific endpoint you want to use. You can test the endpoint using the Test button after configuring the URL.

Request Body (if applicable)​

  1. If your endpoint requires a request body (e.g., for chat completions), configure the request body that will be sent to the OpenAI API. The request body should contain your request parameters formatted as JSON. For chat completion requests, the request body format is typically {"model": "{model_name}", "messages": [{"role": "user", "content": "{your_message}"}]}.

The request body must be valid JSON and must match the OpenAI API's expected format for the specific endpoint you're using. For chat completion endpoints, the request body should include a model property and a messages array. For detailed information about request body formats, see the OpenAI API documentation.

Response Data Path​

  1. Enter the JSONPath expression in the Response Data Path field to specify which part of the API response should be treated as the relevant data by Nexla. For OpenAI API responses, use $.data[*] to extract all items from the data array, or $.choices[*] to extract all choices from a chat completion response, depending on your endpoint.

The JSONPath expression must correctly reference the structure of your OpenAI API response. OpenAI API responses may return data in different structures depending on the endpoint. Ensure your JSONPath expression matches the structure returned by your specific endpoint. The JSONPath expression determines which data will be extracted and processed by Nexla.

Save the Data Source​

  1. Once all of the relevant steps in the above sections have been completed, click the Next button to proceed with the rest of the data flow configuration, or click Save to save the data source configuration for later use.