llorens.marti.works

Color Grading in OpenGL ES 2.0

Recently I've been implementing Color Grading using OpenGL ES 2.0, and I thought it would be constructive to share some thoughts about it.

Color Grading is a process to change an image color with the intention to improve it and/or change the final perception of it. A perfect example would be the movie The Matrix with that underlying green color around all the shots:

Original bottom, Color Graded top.

In the image above (which I found it here), the original pixels have been changed by new ones. One way of doing that change is by using a Look Up Table (LUT). We can use this LUT to input the original pixel and as a result it will return us a new color.

To understand LUTs better, let's present an example. Imagine that we have a LUT of only one dimension, which will translate values from 0.0 to 1.0 into a color. In this example, if we input the value "0.0", we will get the color pure red. If we input the value "1.0", we will get pure white. And if we input the value "0.4", we will get a red-ish color:

With this example in mind, we can use the red channel of a pixel (R) as input values on the LUT. This will end up with a funny final image where:

Otherwise, the pixel gets a gradient between Red and White depending on R.

Of course, only one dimension is not enough for a proper Color Gradient transformation. This is why we need the ability to specify an output color per every possible input color. To do so, we will need a Three-dimensional LUT.

This 3D LUT will allow us to retrieve a color based on an initial input, and we will use the values of the original pixel color as coordinates to access the LUT.

For example, imagine we have an original pixel that is full white. The color values will be R = 1.0, G = 1.0, and B = 1.0. We input those values as a coordinate on the LUT and we get as a result the color stored on the LUT at that coordinate. In this case, let's imagine that the color is purple:

Following with the same example, let's imagine that now we have another original pixel with Green with a bit of Blue (RGB = (0.0, 1.0, 0.33)). If we input that coordinate on the LUT, we will get the stored color on that coordinate, for example Yellow:

Implementation

In OpenGL, there is a concept called 3D Texture, which is an object that allow us to stack many 2D Textures together one on top of the other. Also, it provides something that is very useful, which is the ability to sample the 3D Texture at some given 3D coordinate with correct color interpolation automatically.

In OpenGL ES 2.0 we don't have 3D Textures. So, the first problem is how to represent a 3D Texture with a 2D Texture.

To solve that problem we are going to take every slice of the 3D texture and we are going to put those slices one by the side of the other on a 2D Texture. This means that a 4x4x4 3D Texture will become a 16x4 2D Texture.

The next problem to solve is the difficult one, which is how to sample a 2D Texture like if it was a 3D Texture.

When sampling a 2D Texture to get the correct color, first we need to know which of the two slices are going to contain the final coordinate we want to sample. In the image below we can see that the final coordinate is going to be between Slice 2 and Slice 3.

The way to know both slices is to first start by taking the Z coordinate of the original color (B) and multiply it by the number of slices minus 1. After that, we can find the bottom slice index using the SB = floor(Z) function. In the same way we can find the top slice by using the ST = ceil(Z) function. Finally, we are going to find the value between slices by using the function SM = fract(Z).

After we have those 3 values, the sampling of the code becomes very easy because we only need to sample the bottom slice and the top slice using the same X and Y, but using SB and ST respectively. After that, a simple interpolation between the colors using SM will give us the final color.

Now, let's take a look at the GLSL code and some required adaptations.

First, we need two texture samplers. The first one will contain the original full screen image where we want to apply the color grading. The second one will contain the LUT, in this case a 16x4 2D Texture.

uniform sampler2D tOrig;    // Original Image
uniform sampler2D tLUT;     // LUT Texture

The next step is to sample a color from the Original Image.

vec4 origColor = texture2D(tOrig, uv);

After that we need to sample 2 colors from origColor like we described above, but first we need to compute some offsets and normalized values:

const float slices = 4.0;
const float xOffset = 1.0 / slices;
const float maxSlice = slices - 1.0;
const float npWidth = 1.0 / (slices * slices);
const float npHeight = 1.0 / slices;
const float npHalfWidth = npWidth * 0.5;
const float npHalfHeight = npHeight * 0.5;

It is very important that the coordinate X samples positions inside of the regions of the slices and never on the edges of those slices. If that happens, the color bleed will pollute our final color grading effect. Because of that, here we compute the correct X and Y, alongside with the slices values.

float x = (origColor.r * (xOffset - (npWidth))) + npHalfWidth;
float y = (origColor.g * (1.0 - npHeight)) + npHalfHeight;

float slice = origColor.b * maxSlice;
float SB = floor(slice);
float ST = ceil(slice);
float SM = fract(slice);

Once we have all the offsets computed, we can sample from the 2D Texture and interpolate to get the final color.

vec3 colB = texture2D(tLUT, vec2((xOffset * SB) + x, y)).rgb;
vec3 colT = texture2D(tLUT, vec2((xOffset * ST) + x, y)).rgb;
vec3 colF = mix(colB, colT, SM);

The final step is to output this color to the screen:

gl_FragColor = vec4(colF, 1.0);

With this simple transformation of color, we can apply any color grading to our original image. It needs to be said that if we want better image quality, probably we would need to increase the LUT resolution from 16x4 to something equal or greater than 256x16.

Here is the aggregation of all the code:

uniform sampler2D tOrig;    // Original Image
uniform sampler2D tLUT;     // LUT Texture

vec4 origColor = texture2D(tOrig, uv);

const float slices = 4.0;
const float xOffset = 1.0 / slices;
const float maxSlice = slices - 1.0;
const float npWidth = 1.0 / (slices * slices);
const float npHeight = 1.0 / slices;
const float npHalfWidth = npWidth * 0.5;
const float npHalfHeight = npHeight * 0.5;

float x = (origColor.r * (xOffset - (npWidth))) + npHalfWidth;
float y = (origColor.g * (1.0 - npHeight)) + npHalfHeight;

float slice = origColor.b * maxSlice;
float SB = floor(slice);
float ST = ceil(slice);
float SM = fract(slice);

vec3 colB = texture2D(tLUT, vec2((xOffset * SB) + x, y)).rgb;
vec3 colT = texture2D(tLUT, vec2((xOffset * ST) + x, y)).rgb;
vec3 colF = mix(colB, colT, SM);

gl_FragColor = vec4(colF, 1.0);

I hope it does help in the case anyone needs to have Color Grading implemented on OpenGL ES 2.0 :D

While I was writing this article, I realized I made a mistake. I started from the idea that we needed to sample all the slices with X and Y before using Z. This ended up requiring a lot more texture samples than necessary. I believe that it is good to learn from mistakes, so I will let here the 2nd half of the first draft of the article where we can appreciate the mistake of always start sampling all the slices first :D

...

Because we cannot sample the final color directly, we are going to do it step by step. The first step is to sample on the first 2 dimensions X and Y, which means that we are going to end up with as many samples as slices we have.

In this particular case, we will have 4 colors. Each one of these colors has been sampled on the 16x4 2D Texture, and every sample corresponds to a "local" sample on each of the slices that are stored on the Texture.

From these 4 colors extracted using X and Y, now we need to extract the final color using Z. And here is the tricky part because the sample of the 4 colors was made with the function texture2D(...), but now we have 4 colors and we need to sample by hand.

In this case I asked help of a friend and he exposed to me a very elegant way of doing this last part. The original technique can be seen in this ShaderToy example.

Now, let's take a look at the GLSL code and some required adaptations.

First, we need two texture samplers, the first one will contain the original full screen image where we want to apply the color grading. The second one will contain the LUT, in this case a 16x4 2D texture.

uniform sampler2D tOrig;    // Original Image
uniform sampler2D tLUT;     // LUT Texture

The next step is to sample a color from the Original Image.

vec4 origColor = texture2D(tOrig, uv);

After that, we need to sample the 4 colors using the coordinates X and Y of the origColor. It is very important to have the offsets right, if not, the samples are going to bleed color from nearby slices, this is a very important adaptation here from the ShaderToy example above.

float xOffset = 1.0 / 4.0;
float x = (origColor.r * (xOffset - (1.0 / 16.0))) + (1.0 / 32.0);
float y = (origColor.g * (3.0 / 4.0)) + (1.0 / 8.0);
float z = origColor.b;

It is very important that the coordinate X samples positions inside of the regions of the slices and never on the edges of those slices. If that happens, then color bleed will pollute our final color grading effect.

Also consider that the hardcoded numbers are related to the resolution of the LUT Texture, in this case, 1.0 / 4.0 is because there are 4 slices. 1.0 / 16.0 is because there are 16 pixels along the width of the texture, etc.

After we have the offsets correct, we sample the 4 slices using X and Y.

vec3 col0 = texture2D(tLUT, vec2((xOffset * 0.0) + x, y)).rgb;
vec3 col1 = texture2D(tLUT, vec2((xOffset * 1.0) + x, y)).rgb;
vec3 col2 = texture2D(tLUT, vec2((xOffset * 2.0) + x, y)).rgb;
vec3 col3 = texture2D(tLUT, vec2((xOffset * 3.0) + x, y)).rgb;

Once we have the 4 colors, we need to use Z to compute the final color, which we can do applying what we have learned on the ShaderToy example:

float w = z * 3.0;
vec3 c = col0 * max(1.0 - abs(w - 0.0), 0.0) +
         col1 * max(1.0 - abs(w - 1.0), 0.0) +
         col2 * max(1.0 - abs(w - 2.0), 0.0) +
         col3 * max(1.0 - abs(w - 3.0), 0.0);

The final step is to output this color to the screen:

gl_FragColor = vec4(c, 1.0);

...

   Share Post