Update an image in real-time and rendering it into Three.js

Since its initial steps in 2010, Three.js is one of the most astonishing libraries for 3D developers on the Web. The not-easy-going way to work with WebGL is a good reason to try Three.js (at least for me). Three.js is a cross-browser JavaScript library and API used to create and display interactive 3D computer graphics in a web browser using WebGL 😎

Right now, I am working on a REST-based application that constantly delivers images for clients. Then, I need to display images into an HTML Canvas. Then, I decided to try using Three.js for this. My goal is to change an image placed over a 2D texture into a 3D environment.

via GIPHY

Let's start

To start I used a single 2D plane in Threejs which will be mapped over an image getting from a public API image server.

First, let me write the code for the 101 HTML web page with a canvas graphics component, called threejs-canvas:

<!DOCTYPE html>
<html>
<head>
	<title>Updating quad texture</title>
</head>
<body>
    <canvas id="threejs-canvas"></canvas>
    <script type="module" src="canvas-threejs.js"></script>
</body>
</html>

All my code will be (at least trying to) written in ES6, inside the file canvas-threejs.js. I am assuming that the browser supports module functionality natively. In the js file, I set the following structure:

import * as THREE from 'https://cdnjs.cloudflare.com/ajax/libs/three.js/r125/three.module.min.js';

function main() {...}

main();

First, import the CDN of Threejs taken from https://cdnjs.com/libraries/three.js (for this moment, the last version is r125), create a function main where I will put all the code, and finally invoked it. All the following code should be inside the main function. The ... are not valid, is just to reference the next code 😝

Now, the main code

There are tons of tutorials on how to start with Threejs. A good starting point is the threejs's webpage with several examples: https://threejs.org/examples/. For this setup, I just selected a simple approach: select the canvas component using querySelector (you can use getElementById or others), get the renderer from that canvas, append its DOM element and add an event listener event (I will explain later why).

Finally, in the initial setup, I create the Three.js scene and just one ambient light in white color. As I mentioned, there is good documentation on how and why this works. An excellent explanation of what is a scene is given in the documentation of Three.js. And the code should look like something like:

// all the initialization
const canvas = document.querySelector('#threejs-canvas');
const width = window.innerWidth;
const height = window.innerHeight;
const renderer = new THREE.WebGLRenderer({ canvas });
renderer.setPixelRatio(window.devicePixelRatio);
renderer.setSize(width, height);
document.body.appendChild(renderer.domElement);
document.addEventListener( 'keydown', onKeyDown );

var scene = new THREE.Scene();

//Add ambient light to make the scene more light
const light = new THREE.AmbientLight( 0xffffff ); // soft white light

Great! Until now there is a good starting. Now, I require a place where to obtain images! In my current project images are given from a private server, but for this, it should be from a public server. I choose the Lorem Ipsum for photos: https://picsum.photos/. The website offers public-free images to be requested using a simple URL. Like so, I set images of 256x256 pixels in a random way. Clearly, the random number should represent a positive integer (I do not identify the maximum value), and I randomly selected the number 128:

// url for random picture of square size of 256
const url_base = 'https://picsum.photos/256?random='

// index to access in the picsum as random
let index = 128;
const img_plane = new Image();

Notice, that I created the img_plane variable to store the retrieved image from picsum. Next, I set the source of the image in the URL, plus a random number, plus the extension JPEG. These details are on the piscum webpage. Before that, I set the crossOrigin attribute to an empty string.

img_plane.crossOrigin = "";   // ask for CORS permission
img_plane.src = url_base + index + '.jpg';  // get the image!

When an image from requesting from the internet to be uploaded to the GPU to be used as texture, it could include some issues such as captchas, signatures, private domains, and more. If the image came from another domain that browser will mark the component (canvas in this case) as tainted and you will get a security error when you consult the image:
⚠️ THREE.WebGLState: DOMException: The operation is insecure.

Just to short the story: WebGL (the base of Three.js) bans all images that are not from the same domain; and here is where entering the CORS. CORS means Cross-Origin Resource Sharing which is the way for a webpage to ask the image server permission to use an outsider image. Essentially, you can set three values for the attribute crossOrigin.

undefined: it means do not request permission (default)
anonymous: it means to ask for permission without sending the extra info
use-credentials: it will deliver cookies and other info and the server should decide whether or not it gives permission.
Any other value is equivalent to anonymous (including the empty string)

Let's continue; In Three.js it is necessary to create an object to be added to the scene. In this example, the object will be a plane. The plane will be of Mesh type, and the mesh should be associated with a material. In this case, I just set to a Lambertian Material (that's why I used light in the first instance).

A Material will contain a Texture, and here is where entering the image. Given that pipeline of entities, I have to assure that when I update the image, the Mesh also be updated. The needsUpdate attribute is the key here: it has to be set to true. In this way, it is possible to update the source of the image to update the Texture, and to update update the Material of the Mesh.

Notice that Mesh is composed by the PlaneGeometry (a class for generating plane geometries).
Here the code for this:

// texture variable &activation to update it
const texture_plane = new THREE.Texture(img_plane);
img_plane.onload = () => { texture_plane.needsUpdate = true };

// plane to display
const geometry_plane = new THREE.PlaneGeometry(512, 512, 1);
const mesh_plane = new THREE.Mesh(geometry_plane,
    new THREE.MeshLambertMaterial({ map: texture_plane }))
scene.add(mesh_plane);

// just camera and light
const camera = new THREE.OrthographicCamera(width / - 2, width / 2, 
height / 2, height / - 2, 1, 10);
camera.position.set(0, 0, 1);
scene.add(camera);
scene.add(light);

// rendering function
function render() {
    requestAnimationFrame(render);
    renderer.render(scene, camera);
} render();

The latter details are related to adding a camera to the scene, an orthographic perspective for this example, and the light. Next, the classic render function to real-time displaying (in this case is not fully necessary, but is almost a standard in each graphics application).

Something else?

Remember that before I added the keyDown function? That was added just to test the power of this technique to change images in real-time, this time using the keyboard 🤓

I know there are different ways to do that, but I chose this basic way. The keyCode corresponds to the ASCII code pressed on the keyboard. Values 81 and 87 correspond to q and w keys. On each, a new image is selected from picsum and loaded as the source of the image and thats it!

// using keyboard to update
function onKeyDown(e) {
    switch (e.keyCode) {
        case 81:  //q
            index--;
            console.log(url_base + index + '.jpg');
            img_plane.src = url_base + index + '.jpg';
            break;
        case 87:  //w
            index++;
            console.log(url_base + index + '.jpg');
            img_plane.src = url_base + index + '.jpg';
            break;
        default:
            break;
    }
}

I tested my code using VSCode and using the LiveServer component, see how it looks like: