# Watch out for reduced precision normalize/length in OpenGL ES

Mon

27

Apr 2015

GLSL language for OpenGL ES introduces concept of precision. You can annotate variable declaration (both float and int/uint) with a precision qualifier: `highp`

, `mediump`

or `lowp`

, like:

mediump float a = 3.0;

You can also specify default precision qualifier by using precision statement. Language specification defines minimum required range and precision for each precision qualifier.

`highp`

basically means normal, single-precision, 32-bit float (IEEE 754), as we know it from CPU programming.`mediump`

is said to have have range of at least -2^14 ... 2^14 and relative precision 2^-10, so it can be, for example, implemented using a 16-bit, half-precision float.`lowp`

is said to have range at least -2 ... 2 and absolute precision 2^-8, so basically it can be stored as a 10-bit, fixed-point number.

GPU vendors are free to use more precise data types, or even full 32-bit float for all of them. What exact precision is used depends on specific GPU and maybe even operating system or graphics driver version. Using smaller data types can occupy less memory, calculate faster and consume less battery power. But it comes at the price of reduced precision and range of these numbers. Tom Olson wrote interesting articles about this: "Benchmarking floating-point precision in mobile GPUs": Part I, Part II, Part III.

In this post I'd like to warn you against a specific problem related to it - usage of `length()`

, `normalize()`

and `distance()`

functions. Using smaller data types not only limits precision in terms of number of significant digits, but also available range (over which the value will saturate to -INF/+INF). For mediump, this range is defined as +/-2^14, which is only 16384.

This may still look like a lot, but let's remember that calculating vector length involves intermediate value that it sum of squares of this components. This can grow very big before a square root is applied. For example, for 3D vector:

length(a) = sqrt(a.x*a.x + a.y*a.y + a.z*a.z)

If you do this operation on a mediump vector, the term `a.x*a.x + a.y*a.y + a.z*a.z`

can exceed maximum value for vector as small as (74.0, 74.0, 74.0). It can be very dangerous if you do something like this in your fragment shader:

precision mediump float;

uniform vec3 light_pos;

...

void main()

{

...

vec3 dir_to_light = normalize(pos - light_pos);

// Calculate your lighting and so on...

You might ask: Why isn't this intermediate value stored in high precision before taking its square root to avoid this overflow problem? Obviously it could be, as precision in any place of the shader is free to be higher than the minimum allowed in that place, so some GPU vendors can do it this way, but you shouldn't rely on this. GLSL specification clearly says that the shader is free to use same, reduced precision for intermediate values.

The precision used to internally evaluate an operation, and the precision qualification subsequently associated with any resulting intermediate values, must be at least as high as the highest precision qualification of the operands consumed by the operation.

Conclusion is: When you write shaders for OpenGL ES, watch out for operations that involve calculating vector length (or dot product) like `length()`

, `normalize()`

, `distance()`

, use `highp`

precision for vectors involved and remember that what works on one GPU due to using precision higher than minimum required, may not work on another GPU and it’s still an application issue.

Comments | #math #opengl Share