Skip to main content

Twitter Data Source

Follow the instructions below to create a new data flow that ingests data from a Twitter source in Nexla.
twitter_api.png

Twitter

Create a New Data Flow​

  1. To create a new data flow, navigate to the Integrate section, and click the New Data Flow button. Then, select the desired flow type from the list, and click the Create button.

  2. Select the Twitter connector tile from the list of available connectors. Then, select the credential that will be used to connect to the Twitter instance, and click Next; or, create a new Twitter credential for use in this flow.

  3. In Nexla, Twitter data sources can be created using pre-built endpoint templates, which expedite source setup for common Twitter endpoints. Each template is designed specifically for the corresponding Twitter endpoint, making source configuration easy and efficient.
    • To configure this source using a template, follow the instructions in Configure Using a Template.

Twitter sources can also be configured manually, allowing you to ingest data from Twitter endpoints not included in the pre-built templates or apply further customizations to exactly suit your needs.
• To configure this source manually, follow the instructions in Configure Manually.

Configure Using a Template​

Nexla provides pre-built templates that can be used to rapidly configure data sources to ingest data from common Twitter endpoints. Each template is designed specifically for the corresponding Twitter endpoint, making data source setup easy and efficient.

Endpoint Settings​

  • Select the endpoint from which this source will fetch data from the Endpoint pulldown menu. Available endpoint templates are listed in the expandable boxes below. Click on an endpoint to see more information about it and how to configure your data source for this endpoint.

    Search Tweets Matching Query

    This endpoint template returns a collection of relevant Tweets matching a specified query from your Twitter account. Use this template when you need to search for tweets by keywords, hashtags, or other search criteria for analysis, reporting, or integration purposes.

    • Enter the Search Query in the Search Query field. This should be a UTF-8, URL-encoded search query of 500 characters maximum, including operators. The Search Query determines which tweets will be returned in the search results. You can use Twitter search operators like hashtags, mentions, keywords, and filters.

    This endpoint returns a collection of relevant Tweets matching the specified query from your Twitter account, including tweet content, user information, engagement metrics, and metadata. The endpoint uses next URL pagination to handle large result sets efficiently. Nexla will automatically follow the pagination to fetch subsequent pages of search results.

    For detailed information about tweet search, API response structures, pagination, and available search operators, see the Twitter API documentation.

    Fetch My Tweets and Retweets

    This endpoint template returns a collection of the 200 most recent Tweets and Retweets posted by the authenticating user and the users they follow from your Twitter account. Use this template when you need to access your home timeline, recent tweets, or timeline activity for analysis, reporting, or integration purposes.

    • This endpoint automatically retrieves the 200 most recent Tweets and Retweets from your home timeline. No additional configuration is required beyond selecting this endpoint template. The home timeline is central to how most users interact with the Twitter service.

    This endpoint returns a collection of the 200 most recent Tweets and Retweets posted by the authenticating user and the users they follow from your Twitter account, including tweet content, user information, engagement metrics, and timeline metadata.

    For detailed information about home timelines, API response structures, and available timeline data, see the Twitter API documentation.

    Fetch My Most Recent Mentions

    This endpoint template returns the 200 most recent mentions (Tweets containing a user's @screen_name) for the authenticating user from your Twitter account. Use this template when you need to access mentions, notifications, or engagement activity for analysis, reporting, or integration purposes.

    • This endpoint automatically retrieves the 200 most recent mentions for the authenticating user. No additional configuration is required beyond selecting this endpoint template.

    This endpoint returns the 200 most recent mentions (Tweets containing a user's @screen_name) for the authenticating user from your Twitter account, including mention content, user information, engagement metrics, and mention metadata.

    For detailed information about mentions timelines, API response structures, and available mention data, see the Twitter API documentation.

    Get User Info

    This endpoint template returns a variety of information about the user specified by the screen name from your Twitter account. Use this template when you need to access user profiles, user information, or user metadata for analysis, reporting, or integration purposes.

    • Enter the Screen Name in the Screen Name field. This should be the Twitter screen name (username) of the user whose info you wish to fetch. The Screen Name determines which user's information will be retrieved. The author's most recent Tweet will be returned inline when possible.

    This endpoint returns a variety of information about the user specified by the screen name from your Twitter account, including user profile information, follower counts, tweet counts, and the user's most recent Tweet when possible.

    For detailed information about user information, API response structures, and available user data, see the Twitter API documentation.

    Get User Info from Lookup

    This endpoint template gets user info for Twitter screen names stored in a Nexla lookup table. Use this template when you need to batch fetch user information for multiple screen names from a lookup table for analysis, reporting, or integration purposes.

    • Select the Lookup for fetching Screen Names from the Lookup for fetching Screen Names pulldown menu. This should be the Nexla Lookup that contains Screen Names to iterate over. The Lookup determines which screen names will be used to fetch user information.
    • Enter the Screen Name Column Name in Lookup in the Screen Name Column Name in Lookup field. This should be the column name in the lookup table that contains the Twitter screen names. The Screen Name Column Name determines which column will be used to extract screen names from the lookup table.

    This endpoint gets user info for Twitter screen names stored in a Nexla lookup table, allowing you to batch fetch user information for multiple screen names efficiently. The endpoint iterates over the lookup table and fetches user information for each screen name in the specified column.

    For detailed information about user information, API response structures, and available user data, see the Twitter API documentation.

Endpoint Testing​

Once the selected endpoint template has been configured, Nexla can retrieve a sample of the data that will be fetched according to the current settings. This allows users to verify that the source is configured correctly before saving.

  • To test the current endpoint configuration, click the Test button to the right of the endpoint selection menu. Sample data will be fetched & displayed in the Endpoint Test Result panel on the right.

  • If the sample data is not as expected, review the selected endpoint and associated settings, and make any necessary adjustments. Then, click the Test button again, and check the sample data to ensure that the correct information is displayed.

Configure Manually​

Twitter data sources can be manually configured to ingest data from any valid Twitter API endpoint. Manual configuration provides maximum flexibility for accessing endpoints not covered by pre-built templates or when you need custom API configurations.

With manual configuration, you can also create more complex Twitter sources, such as sources that use chained API calls to fetch data from multiple endpoints or sources that require custom authentication headers or request parameters.

API Method​

  1. To manually configure this source, select the Advanced tab at the top of the configuration screen.

  2. Select the API method that will be used for calls to the Twitter API from the Method pulldown menu. The most common methods are:

    • GET: For retrieving data from the API (most common for Twitter data sources)

API Endpoint URL​

  1. Enter the URL of the Twitter API endpoint from which this source will fetch data in the Set API URL field. This should be the complete URL including the protocol (https://) and any required path parameters.

Ensure the API endpoint URL is correct and accessible with your current credentials. You can test the endpoint using the Test button after configuring the URL.

Path to Data​

If only a subset of the data that will be returned by API endpoint is needed, you can designate the part(s) of the response that should be included in the Nexset(s) produced from this source by specifying the path to the relevant data within the response. This is particularly useful when API responses contain metadata, pagination information, or other data that you don't need for your analysis.

For example, when a request call is used to fetch a list of tweets, the API will typically return an array of tweet records, along with metadata, in the response. By entering the path to the relevant data, you can configure Nexla to treat each element of the returned array as a record.

Path to Data is essential when Twitter API responses have nested structures. Without specifying the correct path, Nexla might not be able to properly parse and organize your data into usable records.

  • To specify which data should be treated as relevant in responses from this source, enter the path to the relevant data in the Set Path to Data in Response field.

    • For responses in JSON format enter the JSON path that points to the object or array that should be treated as relevant data. JSON paths use dot notation (e.g., $.statuses[*] to access an array of tweets within a response object).

    • For responses in XML format, enter the XPath that points to the object/array containing relevant data. XPath uses slash notation (e.g., /response/tweets/tweet to access tweet elements within a tweets element).

    Path to Data Example:

    If the Twitter API response is in JSON format and includes a nested array named statuses that contains the relevant data, the path to the response would be entered as $.statuses[*].

Autogenerate Path Suggestions​

Nexla can also autogenerate data path suggestions based on the response from the API endpoint. These suggested paths can be used as-is or modified to exactly suit your needs.

  • To use this feature, click the Test button next to the Set API URL field to fetch a sample response from the API endpoint. Suggested data paths generated based on the content & format of the response will be displayed in the Suggestions box below the Set Path to Data in Response field.

  • Click on a suggestion to automatically populate the Set Path to Data in Response field with the corresponding path. The populated path can be modified directly within the field if further customization is needed.

    PathSuggestions.png

Endpoint Testing (Manual Configuration)​

After configuring all settings for the selected endpoint, Nexla can retrieve a sample of the data that will be fetched according to the current configuration. This allows users to verify that the source is configured correctly before saving.

  • To test the current endpoint configuration, click the Test button to the right of the endpoint selection menu. Sample data will be fetched & displayed in the Endpoint Test Result panel on the right.

  • If the sample data is not as expected, review the selected endpoint and associated settings, and make any necessary adjustments. Then, click the Test button again, and check the sample data to ensure that the correct information is displayed.

Save & Activate the Source​

  1. Once all of the relevant steps in the above sections have been completed, click the Create button in the upper right corner of the screen to save and create the new Twitter data source. Nexla will now begin ingesting data from the configured endpoint and will organize any data that it finds into one or more Nexsets.