Configuring the HTTP Retriever Data Source

Overview

The HTTP Retriever data source allows Mango by Radix IoT to pull data from external web servers, REST APIs, and other HTTP-accessible endpoints on a configurable schedule. Each poll performs an HTTP GET request, retrieves the response body, and extracts values with regular expressions matched against the raw text.

It works with text-based formats including HTML, XML, plain text, and JSON. For native JSON parsing with JSON Pointer, use the separate HTTP JSON Retriever data source.

Prerequisites

• A running Mango 4.x or 5.x instance
• The URL of the endpoint to poll
• Knowledge of the response format
• An API key or token if the endpoint accepts it in the URL
• Network connectivity from the Mango server to the endpoint

Step 1: Create the Data Source

1. Navigate to Data sources.
2. Click + to add a data source.
3. Select HTTP Retriever.
4. Configure the basic settings:

The unit choices are Milliseconds, Seconds, Minutes and Hours only. If you need a slower or a calendar-aligned schedule, switch the data source to a Cron pattern instead of an update period — the polling section offers one or the other, not both.

Step 2: Configure the URL

Enter the complete endpoint, including the protocol, in URL:

https://api.example.com/v1/sensors/temperature?unit=celsius

Create a separate HTTP Retriever data source for each URL, unless one endpoint returns all required values.

Step 3: Handle Authentication

The HTTP Retriever has no dedicated username/password fields and cannot add custom request headers. Every poll is a plain HTTP GET request. Its available TLS settings control certificate validation, not application authentication.

API Key or Token in the URL

If the API supports query-string credentials, include them in the URL:

https://api.example.com/data?key=abc123def456

Basic Authentication and Custom Headers

The HTTP Retriever cannot directly send HTTP Basic Authentication or custom headers such as Authorization: Bearer ....

If the endpoint requires either:

• Use a Meta data source with a script that performs the authenticated request and feeds the result to a virtual point.
• Put a proxy in front of the endpoint that injects the required header before forwarding the request.

TLS and Certificates

The data source provides connection-level options to verify the server certificate and hostname. You can also trust a specific PKI CA certificate or supply a trusted certificate for self-signed and internal endpoints.

Writing Values Back

The retriever is not strictly read-only. It also carries a Set point URL and Set point name, which Mango requests when a settable point on this data source is written to. The same constraint applies: it is a plain request with no headers, so any credential has to live in that URL. A non-200 response raises a set point failure event rather than failing silently.

Step 4: Set Timeout and Retries

Step 5: Add Data Points with Regex Extraction

1. Click Add points on the data source.
2. Set Name.
3. Set Data type to Numeric, Binary, Multistate, or Alphanumeric as appropriate.
4. Enter a Value RegEx that captures the required value.

Value Capture Group picks which parenthesized group supplies the value. It defaults to 1 — the first capture group — and follows Java numbering, so 2 is the second group. Mango matches with find(), not matches(), so the pattern does not have to consume the whole response.

HTML

For <span class="temp">72.5</span>:

<span class="temp">([\d.]+)</span>

Plain Text

For Temperature: 72.5 F:

Temperature:\s*([\d.]+)

CSV-Like Text

For sensor1,72.5,45.2,normal:

^[^,]+,([\d.]+)

Step 6: Handle JSON Responses

The HTTP Retriever does not parse JSON. It runs regex patterns against the raw JSON response just as it does with any other text.

Given:

{
  "location": "Building A",
  "readings": {
    "temperature": 72.5,
    "humidity": 45.2,
    "status": "normal"
  }
}

This pattern captures the temperature:

"temperature":\s*([\d.]+)

Make the pattern specific enough to avoid matching the wrong field when a key appears more than once.

Step 7: Consider Encoding and Response Size

• Character encoding: Mango uses the charset declared in the response’s Content-Type header. When the header declares none, it falls back to ISO-8859-1, not UTF-8.
• Binary data: HTTP Retriever is for text responses, not images or binary downloads.
• Large responses: The response body is read up to a hard limit of 1 MB. Anything beyond that is discarded.

Step 8: Enable and Verify

1. Save the data source.
2. Enable it with the toggle.
3. Watch the data points for incoming values.
4. Review the data source’s events for connection or parsing errors. Only an HTTP 200 counts as success — any other status code raises a data retrieval failure event.

Verification

• The data source shows an active Status indicator.
• Point values match the source endpoint.
• Value timestamps reflect the configured update period.
• No recurring data-source alarms are present.
• Values change when the source response changes.

Troubleshooting

Connection Refused or Timed Out

• Test the URL with curl from the Mango server.
• Confirm outbound firewall access to the target port, usually 80 or 443.
• For HTTPS, verify the server trusts the remote certificate.

401 or 403 Response

• Confirm the API key or token is valid.
• Verify the API accepts credentials in a URL query parameter.
• If Basic Auth or a custom header is mandatory, use a scripted request or proxy; HTTP Retriever cannot send them.

Regex Does Not Match

• Test against the actual response body.
• Check whitespace, line breaks, escaping, and character encoding.
• Put the desired value inside a capture group.
• Use .*? for non-greedy matching when needed.

Values Are Stale

• Confirm the data source is enabled.
• Check the Update period (its number and unit), or the Cron pattern if the source uses one.
• Review data-source events for recurring poll failures.
• Verify the external endpoint is returning current data.

Related Guides

• Configuring a Modbus TCP Data Source
• Getting Started with the Mango REST API
• Configuring the Meta Data Source for Derived Data Points