Editor’s Note: We're excited to feature Tomoyuki Nakata, a Creative Developer at baqemono, as he shares an impressive tutorial on crafting a mouse-following lens effect using Three.js and GLSL techniques. This comes in light of the upcoming first-ever Three.js conference in Paris, making it a perfect addition to our celebrations! Enjoy the insights!

Concept Overview

Imagine layering a grayscale image alongside a colorful one, accentuated by a square lens effect that tracks the mouse. This tutorial guides you through the steps to build such a captivating mouse-following square lens effect, leveraging the power of Three.js and GLSL.

At its core, this effect pairs two images: a grayscale image forms the base layer, and a vibrant image is revealed inside a square region that follows the mouse around the screen. The area inside the square features lens distortion and a radial RGB shift, while the grayscale image is animated with gentle wave and noise effects.

The final appearance may seem intricate, but the underlying framework is simple, comprising a few key components. We’ll navigate through creating a consistently squared mask that adapts to the viewport dimensions, integrating lens distortions and RGB shifts via fragment shaders, and ensuring smooth animation as the square tracks the mouse. To enhance usability, a graphical user interface (GUI) will also be included for real-time adjustments of various parameters.

This project idea blossomed from a browsing session on Pinterest, where a design caught my eye—an illustration where a grayscale image seemed to be punctuated by a colorful square with a lens effect. Interested in replicating that visual spark with WebGL, I envisioned making the square interactive by allowing it to follow the mouse, and thus, the journey with Three.js and GLSL began.

What You Will Create

The final effect encapsulates multiple elements:

• A full-screen grayscale image
• A color image revealed solely within the square
• A lens distortion effect that bulges outward from the square's center
• A radial RGB shift becoming more pronounced near the square's edges
• A square mask that maintains its shape and dimensions despite screen size variations
• A fluid mouse tracking interaction with a slight responsiveness delay
• Subtle wave and distortion effects applied to the grayscale image
• A GUI to tweak parameters on-the-fly

Despite the apparent complexity of the outcome, this project avoids the use of complicated post-processing techniques or the incorporation of 3D models. Instead, we craft the entire effect exclusively within a fragment shader by blending these components.

File Organization

The project structure is logically set up to streamline development:

src/
└── scripts/
    ├── webgl/
    │   ├── glsl/
    │   │   ├── chunks/
    │   │   │   ├── ccLens.glsl
    │   │   │   ├── coverUv.glsl
    │   │   │   └── random3.glsl
    │   │   ├── frag/
    │   │   │   └── frag.glsl
    │   │   └── vert/
    │   │       └── vert.glsl
    │   ├── mesh/
    │   │   └── Mesh.ts
    │   ├── stage/
    │   │   └── Stage.ts
    │   └── Webgl.ts
    └── index.ts
    

• glsl: chunks encompasses reusable functions; frag contains fragment shaders, while vert is for vertex shaders.
• mesh (Mesh): Focuses on window sizing, texture loading, mesh creation, and updating uniforms.
• stage (Stage): Oversees scene management, camera and renderer initialization, along with essential updates.
• Webgl: Responsible for instantiating Mesh and Stage, connecting the GUI, managing events, and running the render loop.

The class structure aligns with typical practices, allowing us to concentrate primarily on the intricacies of the fragment shader throughout this guide.

Setting Up the Scene

To kick things off, we’ll explore the Stage class. This component sets up the scene, including the camera and renderer, along with other fundamental tools necessary for working in the Three.js environment. While much of the setup adheres to common conventions, it’s crucial to examine how we determine the camera’s Z position.

Determining the Camera Z Position

The camera's Z position is calculated through a function named calcViewportDistance.

const calcViewportDistance = (height: number, fov: number): number => {
  return height / (2 * Math.tan((fov * Math.PI) / 360))
}

In essence, this calculation finds the necessary distance at which the height visible to the camera aligns with the specified height. By scaling a 1 x 1 plane to match the viewport's width and height, we ensure that the mesh fits perfectly within the camera's field of view (FOV).

Exploring the Mesh Class

Next, let's turn our attention to the Mesh class. This component is responsible for everything mesh-related, from handling the window size to texture loading and mesh creation with both geometry and material. Its structure is relatively straightforward, so our focus will shift to progressively building the final visual through shader development.

Understanding the Vertex Shader

Given that this effect doesn’t require vertex deformation, the vertex shader remains uncomplicated. Its primary role is to transfer the geometry’s UV coordinates to the fragment shader while adjusting the plane’s vertex positions to align with the screen-space coordinates, taking into account the mesh scale, camera position, and FOV.

precision highp float;

attribute vec3 position;
attribute vec2 uv;

uniform mat4 modelViewMatrix;
uniform mat4 projectionMatrix;

varying vec2 v_uv;

void main() {
  v_uv = uv;
  
  gl_Position = projectionMatrix * modelViewMatrix * vec4(position, 1.0);
}

Diving Into the Fragment Shader

Displaying Textures in Fullscreen

Now, we pivot to the fragment shader where our goal is to render the two images across the entire screen. We initiate this process inside the Mesh class's setTexture method. texture1 serves as the color image, while texture2 is the grayscale image covering the remainder.

const loader = new TextureLoader();
const texture1Path = this.$target.dataset.texture1Path;
const texture2Path = this.$target.dataset.texture2Path;

if (!texture1Path || !texture2Path) return;

const [texture1, texture2] = await Promise.all([
  loader.loadAsync(texture1Path),
  loader.loadAsync(texture2Path)
]);

this.uniforms.u_texture1.value = texture1;
this.uniforms.u_texture2.value = texture2;
this.uniforms.u_textureSize1.value.set(
  texture1.image.width,
  texture1.image.height
);
this.uniforms.u_textureSize2.value.set(
  texture2.image.width,
  texture2.image.height
);

Within the fragment shader, we sample colors from both textures, setting the stage for our initial rendering. As both images share the same aspect ratio, we could theoretically combine their texture size calculations, but we opt to keep them separate to accommodate for images with varying aspect ratios.

precision highp float;

uniform sampler2D u_texture1;
uniform sampler2D u_texture2;
uniform vec2 u_meshSize;
uniform vec2 u_textureSize1;
uniform vec2 u_textureSize2;

varying vec2 v_uv;

#include "../chunks/coverUv.glsl"

void main() {
  vec2 texture1Uv = getCoverUv(v_uv, u_meshSize, u_textureSize1);
  vec2 texture2Uv = getCoverUv(v_uv, u_meshSize, u_textureSize2);

  vec4 insideColor = texture2D(u_texture1, texture1Uv);
  vec4 outsideColor = texture2D(u_texture2, texture2Uv);

  gl_FragColor = insideColor;
}

The getCoverUv function ensures that images maintain their aspect ratio, preventing any distortion during the display process. Instead of relying solely on v_uv, we create UV coordinates that will crop images from the center, akin to how CSS background properties function by using background-size: cover.

vec2 getCoverUv(vec2 uv, vec2 meshSize, vec2 textureSize) {
  vec2 meshRatio = vec2(meshSize.x / meshSize.y, meshSize.y / meshSize.x);
  vec2 textureRatio = vec2(textureSize.x / textureSize.y, textureSize.y / textureSize.x);

  vec2 resolutionRatio = vec2(
    min(meshRatio.x / textureRatio.x, 1.0),
    min(meshRatio.y / textureRatio.y, 1.0)
  );

  return (uv - 0.5) * resolutionRatio + 0.5;
}

With this setup, the color image fills the screen as intended, as displayed in the subsequent figure.

You can find the initial code outlined in 01-display-color-image.glsl.

Once the color image is set, we can adjust the final line to display the grayscale image, verifying its successful rendering.

gl_FragColor = outsideColor;

You can find this segment of the code at 02-display-grayscale-image.glsl.

For clarity, the color and grayscale image variables are named insideColor and outsideColor, which helps illustrate their respective roles in the final composite.

Coordinate System for the Square

Next, we’ll create the mask that delineates the transition between the grayscale and color images. Transforming v_uv into the range of -1.0 to 1.0 allows us to establish a more manageable coordinate system centered on the screen.

vec2 uvSquare = v_uv * 2.0 - 1.0;

Creating the Square Mask

Using the uvSquare coordinates established earlier, we can now form the square mask. By designating u_squareSize to represent half the length of one side of the square, we define the boundaries—left, right, bottom, and top—of the shape.

uniform float u_squareSize;

float squareHalfSize = u_squareSize;
float left = -squareHalfSize;
float right = squareHalfSize;
float bottom = -squareHalfSize;
float top = squareHalfSize;

We employ the step function to check whether each pixel falls within the defined boundaries, multiplying the resulting boolean values. This yields 1.0 within the square, effectively identifying the interior, while returning 0.0 elsewhere.

float squareMask =
  step(left, uvSquare.x) *
  (1.0 - step(right, uvSquare.x)) *
  step(bottom, uvSquare.y) *
  (1.0 - step(top, uvSquare.y));

To visualize the mask shape, we can output the squareMask value as a color directly.

gl_FragColor = vec4(vec3(squareMask), 1.0);

In doing so, we should see a white rectangle at the center, indicating the mask’s defined area.

Refer to the code for this stage in 03-create-square-mask.glsl.

Adjusting for Aspect Ratio

Currently, we have a rectangular mask, but it will distort in aspect ratio if the viewport is resized. To remedy this, we calculate an aspect-ratio correction factor based on the mesh dimensions determining the square's boundaries.

vec2 squareAspectScale = vec2(
  min(u_meshSize.y / u_meshSize.x, 1.0),
  min(u_meshSize.x / u_meshSize.y, 1.0)
);

This factor is then used to adjust the coordinates involved in the square's test, ensuring it retains a proper shape regardless of screen dimensions.

uvSquare /= squareAspectScale;

Check the code here: 04-correct-mask-aspect-ratio.glsl.

Blending the Two Images

Using the resulting squareMask, we can blend both images. Where the mask is 0.0, we will display outsideColor; where it's 1.0, the insideColor takes over. This creates a visually engaging square crop of the colorful image overlaying the grayscale backdrop.

vec4 finalColor = mix(outsideColor, insideColor, squareMask);

gl_FragColor = finalColor;

You can find the code at this point in 05-composite-images.glsl.

Incorporating CC Lens Distortion