How to Build Data Validation Scripts

Overview

Bad data from sensors—spikes, dropouts, error codes, stuck values—can corrupt historical records and trigger false alarms. Data validation scripts catch these problems before values are stored, keeping time-series data clean and alarms reliable.

Prerequisites

• A running Mango by Radix IoT v4.x or v5.x instance
• Access to edit data points, create meta points, and configure global scripts
• Known valid ranges and fault codes for the sensors being monitored
• Enough historical data to choose realistic validation thresholds

Approach 1: Point Configuration Without Scripting

Mango’s built-in Discard extreme values setting handles simple range validation:

1. Open the data point’s edit page.
2. Enable Discard extreme values.
3. Set the Discard low limit and Discard high limit.
4. Save the point.

Values outside the configured range are silently discarded: they are not logged or displayed. This is the simplest approach and should be the first line of defense.

Approach 2: Meta Point Validation

For logic more complex than range checks, create a meta point that reads the raw source point and produces a validated value.

Clamp Values to a Valid Range

// Context: raw = raw temperature point
var value = raw.value;

if (value < -40) value = -40;
if (value > 300) value = 300;

return value;

Filter Known Error Codes

// Context: raw = source point
var errorCodes = [-999, -1, 65535, 32767];

if (errorCodes.indexOf(raw.value) >= 0) {
    return UNCHANGED; // Leave the meta point's current value in place
}

return raw.value;

Detect Spikes

// Reject values that change more than 50 units in one poll
// Context: raw = source point
var current = raw.value;
var previous = raw.ago(SECOND, 1);

if (previous !== null) {
    var diff = Math.abs(current - previous);

    if (diff > 50) {
        LOG.warn(
            "Spike detected on " +
            raw.dataPointWrapper.name +
            ": " +
            current
        );
        return UNCHANGED;
    }
}

return current;

Detect a Stuck Value

// Context: raw = source point
var to = raw.time;
var from = to - (2 * 60 * 60 * 1000);
var history = raw.pointValuesBetween(from, to);

if (history.size() > 10) {
    var allSame = true;
    var first = history.get(0).doubleValue;

    for (var i = 1; i < history.size(); i++) {
        if (history.get(i).doubleValue !== first) {
            allSame = false;
            break;
        }
    }

    if (allSame) {
        LOG.warn(
            "Stuck value detected on " +
            raw.dataPointWrapper.name +
            ": " +
            raw.value
        );
        return -1; // Multistate code for "suspect"
    }
}

return 0; // Multistate code for "normal"

Approach 3: Global Script for Batch Validation

A scheduled global script can review recently logged values across many points and flag anomalies:

// Nightly validation: check all temperature points for spikes
var points = DataPointQuery.query("eq(tags.pointType,temperature)");
var to = new Date().getTime();
var from = to - (24 * 60 * 60 * 1000);

for (var i = 0; i < points.size(); i++) {
    var pt = points.get(i);

    if (pt.runtime === null) continue;

    var values = pt.runtime.pointValuesBetween(from, to);
    var min = Infinity;
    var max = -Infinity;

    for (var j = 0; j < values.size(); j++) {
        var value = values.get(j).doubleValue;
        if (value < min) min = value;
        if (value > max) max = value;
    }

    if (max - min > 100) {
        LOG.warn(
            "Possible spike on " +
            pt.name +
            ": range = " +
            min +
            " to " +
            max
        );
    }
}

Use DataPointQuery.query(rql) with an RQL expression such as eq(tags.pointType,temperature). Each result is a data point wrapper. Its .runtime property is null when the point is not enabled; otherwise, it exposes pointValuesBetween(from, to) for retrieving history.

Building a Validation Strategy

• Enable Discard extreme values on numeric points as the first line of defense.
• Add meta point validation where you need spike detection, error-code filtering, or more complex logic.
• Schedule a global script for periodic batch validation.
• Log validation events so recurring data-quality problems can be traced to their source.

Verification

• Confirm an in-range source value reaches the validated point unchanged.
• Submit a known error code and verify the meta point retains its previous value.
• Submit an artificial spike and confirm the script logs a warning containing raw.dataPointWrapper.name.
• Confirm stuck-value detection uses samples returned by raw.pointValuesBetween(from, to).
• Review historical data to ensure rejected values were not stored as 0, false, or an empty string.

Troubleshooting

Rejected Values Become Zero or False

Replace return null with return UNCHANGED. A null result can be coerced to the point data type’s default value.

Spike Detection Fails or Reports Undefined

• Use raw.ago(periodType, count) for one earlier value.
• Use raw.dataPointWrapper.name for the point name.
• Do not treat raw.past() as an array.

Stuck-Value Detection Throws an Error

• Retrieve samples with raw.pointValuesBetween(from, to).
• Iterate with .size() and .get(i) — the result is a Java list, and .length is
  undefined on Nashorn, which makes the loop silently skip.
• Compare each sample’s .doubleValue, not its boxed .value.
• Confirm the time range contains enough logged samples.

Global Script Returns No Points

• Confirm the points have the expected tags.
• Use DataPointQuery.query("eq(tags.pointType,temperature)").
• Iterate the result with .size() and .get(i). It is a Java list, so points.length
  can be undefined and the loop will report zero points without raising an error.
• Skip results whose .runtime property is null.

Related Guides

• Guide to Scripting Locations in Mango
• How to Create Unit Conversions with Meta Points
• How to Use Global Scripts for Reusable Functions
• How to Configure Point Logging for Optimal Data Collection