How to integrate AI facial expression detection to control your video – Javascript
How to integrate AI facial expression detection to control your video – Javascript
Learn how to add AI facial expression detection to your videos using Javascript.
Yohann Martzolff
June 24, 2024
Today, with the rise of AI, new possibilities are emerging that allow us to take video control to the next level. One such innovative feature is AI facial expression detection. By integrating this technology into your videos, you can create a more engaging and responsive user experience.
In this blog, we will learn how to integrate AI facial expression detection into your videos using Javascript.
We use JavaScript here because it can be easily integrated into virtually any web application. Additionally, its adaptability to different platforms and its ability to interact with HTML and CSS empower you to create rich, interactive web experiences that cater to diverse user needs.
To learn how to do the integration with React, head to this blog.
1. Build the application’s structure
Start by creating a simple index.html file
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>Face landmark detection</title>
<script src="index.js" type="module"></script>
</head>
<body>
<h1>My facial expression detection app</h1>
</body>
</html>
And an index.js file
async function run() {
alert("Hello World!");
}
document.addEventListener("DOMContentLoaded", run);
Start your application by running the npx http-server command and open the dedicated http://localhost port to see your application running. A “Hello World!” alert should appear.
Note: If you don’t have the npx package already installed on your computer, you can install it by running npm i -g npx in a terminal.
2. Add the external JS scripts
Import the MediaPipe script by adding these lines at the top of index.js
import vision from "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.3";
const { FaceLandmarker, FilesetResolver } = vision;
Import the api.video-player-sdk script by adding the following
<head>
...
<script src="https://unpkg.com/@api.video/player-sdk" defer></script>
</head>
Note: We’re using the api.video’s player in this article, but feel free to use your favorite one and adapt the callbacks to it.
3. Add a video player
Add a <div /> tag to index.html. It will be targeted by the api.video player SDK to display the video you're going to control with your facial expressions.
<div id="video" style="height: 380px"></div>
Target the element in index.js with the following code.
let player;
async function run() {
player = new PlayerSdk(
"#video", // Target the HTLM element with the id "video"
{
id: "vi5fv44Hol1jFrCovyktAJS9", // Feel free to change this video id with one of yours
muted: true, // We need the video to be muted to be able to control it without document interaction
}
);
}
Run your application with npx http-server and you should see a video that you can play, pause, etc...
4. Record your facial expressions
To control the player with your facial expressions, you need to stream your face through your webcam and pass this streamed data to the trained machine-learning model.
Add a <video /> tag to index.html and apply a display: "none" inline style to hide the element, and an autoplay attribute so the webcam stream time is updated.
<video id="webcam" style="display: none" autoplay></video>
5. Load the machine-learning model
Before we can use the machine-learning model, we must wait for it to finish loading. These models can be large and take a moment to get everything needed to run.
Add and run a new function in index.js
async function createFaceLandmarker() {
const filesetResolver = await FilesetResolver.forVisionTasks(
"https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.3/wasm"
);
faceLandmarker = await FaceLandmarker.createFromOptions(filesetResolver, {
baseOptions: {
modelAssetPath: `https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.task`,
delegate: "GPU",
},
outputFaceBlendshapes: true,
runningMode: "VIDEO",
numFaces: 1,
});
}
async function run() {
player = new PlayerSdk(
"#video", // Target the HTLM element with the id "video"
{
id: "vi5fv44Hol1jFrCovyktAJS9", // Feel free to change this video id with one of yours
muted: true, // We need the video to be muted to be able to control it without document interaction
}
);
await createFaceLandmarker();
}
6. Start detecting the webcam stream
Once the machine-learning model has been loaded and the faceLandmarker class has been instantiated, you can start detecting the stream.
Add the following code to index.js
const webcam = document.getElementById("webcam");
async function predictWebcam() {
let lastVideoTime = -1;
let results = undefined;
let startTimeMs = performance.now();
if (lastVideoTime !== webcam.currentTime) {
lastVideoTime = webcam.currentTime;
results = faceLandmarker.detectForVideo(webcam, startTimeMs);
}
if (results?.faceBlendshapes && results.faceBlendshapes[0]) {
console.log(results.faceBlendshapes[0].categories);
}
window.requestAnimationFrame(predictWebcam);
}
async function run() {
...
if (
faceLandmarker &&
!!(navigator.mediaDevices && !!navigator.mediaDevices.getUserMedia) &&
webcam &&
player
) {
navigator.mediaDevices.getUserMedia({ video: true }).then((stream) => {
webcam.srcObject = stream;
webcam.addEventListener("loadeddata", predictWebcam);
});
}
}
Check your application, and open your browser console. You should see a lot of console logs appearing each millisecond.
Each one of them logs an array of objects of type:
{
"categoryName": "string",
"displayName": "string",
"index": "number",
"score": "number"
}
7. Use the facial expression detection results
You now have access to a list of facial expression detection results. Use them to do whatever you want and in this case, to control your video player.
Let’s assume that you want to play the video when you smile, and pause it when you do a kiss.
Check the facial expression detection results on each frame and play or pause your video depending on the score of the wanted result by replacing your console.log with the following.
async function predictWebcam() {
...
if (results?.faceBlendshapes && results.faceBlendshapes[0]) {
// PLAY
const smile = results.faceBlendshapes[0].categories.find(
(shape) =>
shape.categoryName === "mouthSmileLeft" ||
shape.displayName === "mouthSmileLeft"
)?.score;
if (smile && smile > 0.6) player.play();
// PAUSE
const kiss = results.faceBlendshapes[0].categories.find(
(shape) =>
shape.categoryName === "mouthPucker" ||
shape.displayName === "mouthPucker"
)?.score;
if (kiss && kiss > 0.6) player.pause();
}
...
}
Check your application, and try to smile at your webcam. It should trigger the player’s play callback and play the video. Then, try to do a kiss (😗) in front of your webcam, and the video should pause.
8. Display the results
It can be helpful to have a visual representation of the detection results.
Display these results as a list and highlight the detected facial expressions by adding a <ul /> tag to index.html
<body>
...
<ul
id="results"
style="
display: grid;
grid-template-columns: repeat(5, 1fr);
column-gap: 20px;
"
/>
</body>
And the code below to index.js
const detectionResults = document.getElementById("detection-results");
async function predictWebcam() {
...
if (results?.faceBlendshapes && results.faceBlendshapes[0]) {
// DISPLAY
let htmlMaker = "";
results.faceBlendshapes[0].categories.forEach((shape) => {
htmlMaker += `
<li class="blend-shapes-item" style="${
shape.score > 0.6 ? "background: green" : ""
}">
<span class="blend-shapes-label">${
shape.displayName || shape.categoryName
}</span>
<span class="blend-shapes-value" style="width: calc(${
+shape.score * 100
}% - 120px)">${(+shape.score).toFixed(4)}</span>
</li>
`;
});
if (detectionResults) detectionResults.innerHTML = htmlMaker;
...
}
...
}
Now, you should see a list of results displayed in your application, with a green highlight on the ones with more than 60% confidence.
9. Code samples
Check the full index.html and index.js codes below
index.html
<!-- index.html -->
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>Face landmark detection</title>
<script src="index.js" type="module"></script>
<script src="https://unpkg.com/@api.video/player-sdk" defer></script>
</head>
<body>
<h1>My facial expression detection app</h1>
<div id="video" style="height: 380px"></div>
<video id="webcam" style="display: none" autoplay></video>
<ul
id="detection-results"
style="
display: grid;
grid-template-columns: repeat(5, 1fr);
column-gap: 20px;
"
/>
</body>
</html>
index.js
// MediaPipe
import vision from "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.3";
const { FaceLandmarker, FilesetResolver } = vision;
let player;
let faceLandmarker;
async function createFaceLandmarker() {
const filesetResolver = await FilesetResolver.forVisionTasks(
"https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.3/wasm"
);
faceLandmarker = await FaceLandmarker.createFromOptions(filesetResolver, {
baseOptions: {
modelAssetPath: `https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.task`,
delegate: "GPU",
},
outputFaceBlendshapes: true,
runningMode: "VIDEO",
numFaces: 1,
});
}
const webcam = document.getElementById("webcam");
const detectionResults = document.getElementById("detection-results");
async function predictWebcam() {
let lastVideoTime = -1;
let results = undefined;
let startTimeMs = performance.now();
if (lastVideoTime !== webcam.currentTime) {
lastVideoTime = webcam.currentTime;
results = faceLandmarker.detectForVideo(webcam, startTimeMs);
}
if (results?.faceBlendshapes && results.faceBlendshapes[0]) {
// DISPLAY
let htmlMaker = "";
results.faceBlendshapes[0].categories.forEach((shape) => {
htmlMaker += `
<li class="blend-shapes-item" style="$ {
shape.score > 0.6 ? "background: green" : ""
}">
<span class="blend-shapes-label">$ {
shape.displayName || shape.categoryName
}</span>
<span class="blend-shapes-value" style="width: calc($ {
+shape.score * 100
}% - 120px)">${(+shape.score).toFixed(4)}</span>
</li>
`;
});
if (detectionResults) detectionResults.innerHTML = htmlMaker;
// PLAY
const smile = results.faceBlendshapes[0].categories.find(
(shape) =>
shape.categoryName === "mouthSmileLeft" ||
shape.displayName === "mouthSmileLeft"
)?.score;
if (smile && smile > 0.6) player.play();
// PAUSE
const kiss = results.faceBlendshapes[0].categories.find(
(shape) =>
shape.categoryName === "mouthPucker" ||
shape.displayName === "mouthPucker"
)?.score;
if (kiss && kiss > 0.6) player.pause();
}
window.requestAnimationFrame(predictWebcam);
}
async function run() {
player = new PlayerSdk("#video", {
id: "vi5fv44Hol1jFrCovyktAJS9",
muted: true,
});
await createFaceLandmarker();
if (
faceLandmarker &&
!!(navigator.mediaDevices && !!navigator.mediaDevices.getUserMedia) &&
webcam &&
player
) {
navigator.mediaDevices.getUserMedia({ video: true }).then((stream) => {
webcam.srcObject = stream;
webcam.addEventListener("loadeddata", predictWebcam);
});
}
}
document.addEventListener("DOMContentLoaded", run);
We hope this blog helped you understand the coding behind integrating AI facial expression detection to your videos.
For any questions or doubts you may have, chat with us on our website. If you would like to try out api.video for your videos, sign-up for a free sandbox account here.