Transfer Characteristic
Deviation of a sensor’s actual output from a theoretical straight line represents a systematic error that must be accounted for in the measurement system. Measuring non-linearity involves comparing the actual readings at multiple points across the full scale to a reference line. This error is often the largest component of the total accuracy specification.
Least Squares
Calculation of the best fit line through the data points provides a standard way to quantify the average deviation. The non-linearity is then defined as the maximum distance between any single data point and this calculated line. Engineers use this value to determine if a simple gain and offset adjustment is sufficient or if a more complex polynomial correction is required.
Signal Correction
Digital signal processing can remove much of this error by applying a mathematical function to the raw output of the sensor. The non-linearity is reduced by using a lookup table or a curve fitting algorithm that compensates for the known curvature of the sensor response. This process happens inside the local controller of the instrument.
Error Contribution
High precision applications require the error to be minimized to prevent bias in the final results. Even after correction, some residual non-linearity remains due to the limits of the calculation.