Putting Video on Canvas

On the desktop, you can display video on canvas by calling drawImage() using a video element as the image source. The frame currently under the playhead is rendered on the canvas. If you call the play() method on the video, and call drawImage() at frequent intervals, the video is played back smoothly on the canvas. You can play a video without displaying it outside of the canvas by setting the video element’s display attribute to none.

Using video as a source for drawImage() involves a lot of system resources. Generally speaking, video is best displayed using the video element, not the canvas element. To composite canvas text or animations over moving video, it’s better to use a video element behind the canvas—the video shows through the transparent background of the canvas without the overhead of displaying video on the canvas itself.

On iOS-based devices with small screens—such as iPhone and iPod touch—video always plays in fullscreen mode, so the canvas cannot be superimposed on playing video. On iOS-based devices with larger screens, such as iPad, you can superimpose canvas graphics on playing video, just as you can on the desktop.

Use video as an image source for the canvas only when you need to access the pixel data of a video. Video on canvas is useful for things such as realtime image processing—green screen effects, extracting the average color value from a video frame, capturing a series of stills from a video—or special effects such as putting segments of a moving video image on tiles and moving them independently.

Canvas Over Video

When you want to superimpose canvas graphics or animation on a video, the best approach is to position a canvas element on top of a video element, so the video shows through the transparent parts of the canvas. The example in Listing 17-1 plays an MPEG-4 video stream behind the canvas, with spinning text overlaid on the video. A simple play/pause button controls the video, and the animation is tied to the video. The output is illustrated in Figure 17-1.

Figure 17-1  Video under canvas


Listing 17-1  Displaying video behind the canvas

<html>
<head>
    <meta name = "viewport" content = "width = device-width">
    <title>Canvas Over Video</title>
    <script type="text/javascript">
        var can, ctx, vid, playButton,
            vidTimer, text, angle = 0;
 
        function init() {
            vid = document.getElementById("vid");
            playButton = document.getElementById("play");
            can = document.getElementById("can");
            ctx = can.getContext('2d');
            vid.addEventListener('ended', vidEnd, false);
            vid.addEventListener('play', setAnimate, false);
            ctx.fillStyle = "white";
            ctx.strokeStyle = "black";
            ctx.font = "20pt Helvetica";
            ctx.textAlign = "center";
            ctx.textBaseline = "middle" ;
            animate();
        }
 
        // Draw spinning text
        function animate() {
            ctx.save();
            // clear screen
            ctx.clearRect(0, 0,can.width, can.height);
            // set origin at center of canvas
            ctx.translate(can.width / 2, can.height / 2);
            // spin the canvas
            ctx.rotate(angle);
            // draw white text with a black outline
            ctx.fillText ("Animation over video!", 0, 0);
            ctx.strokeText ("Animation over video!", 0, 0);
            ctx.restore();
            // increment rotation angle, reset to 0 at 2 * Pi
            angle += 0.1;
            if (angle > Math.PI * 2)
                angle = 0;
            // if video not paused or ended, repeat in 50 msec
            var vidState = document.querySelector("video");
            if (!vidState.paused && !vidState.ended)
                vidTimer = setTimeout(animate, 50);
        }
 
        // Start animation when video starts playing
        function setAnimate() {
            clearTimeout(vidTimer);
            vidTimer = setTimeout(animate, 50);
        }
 
        function playVideo() {
            if (playButton.value == "Play") {
                vid.play();
                play.value = "Pause";
            }
            else {
                vid.pause();
                playButton.value = "Play";
            }
        }
 
        function vidEnd() {
            playButton.value = "Play";
            clearTimeout(vidTimer);
        }
    </script>
</head>
<body onload="init()">
    <video src="assets/myMovie.m4v" id="vid"
           style="position:absolute; top: 10; left: 10;">
    </video>
    <canvas id="can" height="270" width="480"
            style="position:absolute; top: 10; left: 10;">
    </canvas>
    <p style="position:absolute; top: 300; left: 50;">
        <input id="play" type="button" value="Play" onclick="playVideo()"
               style="font:18pt Helvetica" />
    </p>
</body>
</html>

Tiled Video

There are a number of effects you can implement by displaying a video as a collection of independent tiles, such as explosions, particle effects, and moving jigsaw puzzles. You map parts of a video onto tiles by calling drawImage(sourceX,sourceY,sourceWidth,sourceHeight, destX,destY,destWidth,destHeight). See “Drawing an Image with Region Mapping” for details.

The example in Listing 17-2 divides a video image into four tiles and rotates each tile independently, as illustrated in Figure 17-2.

Figure 17-2  Rotating video tiles


Listing 17-2  Breaking video into tiles

<html>
<head>
    <meta name="viewport" content="width=device-width" />
    <title>Tiled Video</title>
    <script type="text/javascript">
        var can, ctx, vid, playButton, vidTimer,
            angle = -1, imageWidth = 480, imageHeight = 270,
            tWide, tHigh, xOffset, yOffset;
            topOffset = 72, leftOffset = 22, vidState;
 
        function init() {
            vid = document.getElementById("vid");
            vidState = document.querySelector("video");
            playButton = document.getElementById("play");
            can = document.getElementById("can");
            ctx = can.getContext('2d');
            vid.addEventListener('ended', vidEnd, false);
            vid.addEventListener('play', setAnimate, false);
            // Tile width and height (divide image into 4 tiles)
            tWide = imageWidth / 2;
            tHigh = imageHeight / 2;
            // Offset from center of tile image to upper left corner
            xOffset = imageWidth / -4;
            yOffset = imageHeight / -4;
            animate();
        }
 
        // If video not paused, draw image in 4 tiles,
        // rotating each tile about the center
        function animate() {
            if (!vidState.paused) {
                // If video ended,draw image once with no rotation
                if (vidState.ended) angle = 0;
                // If angle is negative, increment but display without rotation
                var dispAngle = Math.max(0,angle);
                // Clear screen
                ctx.clearRect(0,0,can.width,can.height);
                // Draw four tiles, rotated about the center
                ctx.save();
                  ctx.translate(tWide / 2 + leftOffset, tHigh / 2 + topOffset);
                  ctx.rotate(dispAngle);
                ctx.drawImage(vid,0,0,tWide,tHigh,xOffset,yOffset,tWide,tHigh);
                ctx.restore();
                ctx.save();
                  ctx.translate(tWide * 3 / 2 + leftOffset, tHigh / 2 + topOffset);
                  ctx.rotate(dispAngle);
                ctx.drawImage(vid,tWide,0,tWide,tHigh,xOffset,yOffset,tWide,tHigh);
                ctx.restore();
                ctx.save();
                  ctx.translate(tWide / 2 + leftOffset, tHigh * 3 / 2 + topOffset);
                  ctx.rotate(dispAngle);
                ctx.drawImage(vid,0,tHigh,tWide,tHigh,xOffset,yOffset,tWide,tHigh);
                ctx.restore();
                ctx.save();
                  ctx.translate(tWide * 3 / 2 + leftOffset, tHigh * 3 / 2 + topOffset);
                  ctx.rotate(dispAngle);
                ctx.drawImage(vid,tWide,tHigh,tWide,tHigh,xOffset,yOffset,tWide,tHigh);
                ctx.restore();
                // Increment angle
                angle += 0.02;
                // If at full circle, reset angle
                // and display without rotation for fifty cycles
                if (angle > Math.PI * 2)
                    angle = -1;
                // If video not paused (and video not ended), repeat animation in 50 ms
                if (!vidState.ended)
                    vidTimer = setTimeout(animate, 50);
            }
        }
 
        // Set display angle to 0 for fifty cycles
        function reset() {
            angle = -1;
        }
 
        // Start animation when video starts playing
        function setAnimate() {
            clearTimeout(vidTimer);
            vidTimer = setTimeout(animate, 50);
        }
 
 
        function playVideo() {
            if (playButton.value == "Play") {
                vid.play();
                play.value = "Pause";
            }
            else {
                vid.pause();
                playButton.value = "Play";
            }
        }
 
        function vidEnd() {
            playButton.value = "Play";
        }
    </script>
</head>
<body onload="init()">
    <video id="vid" src="assets/myMovie.m4v" style="display:none"></video>
    <canvas id="can" height="420" width="522">
    </canvas>
    <hr />
    <p>
        <input type="button" value="Play" onclick="playVideo()" id="play"
               style="font:18pt Helvetica">
        <input type="button" value="Reset" onclick="reset()" style="font:18pt Helvetica">
    </p>
</body>
</html>

Image Processing

It’s possible to do realtime image processing on video by displaying the video on canvas and manipulating the pixels of the canvas bitmap. The canvas bitmap is accessed by calling getImageData(x, y, width, height) and accessing the imageData object’s data property, which is a pixel array. See “Pixel Manipulation” for details.

The example in Listing 17-3 displays a video stream on the canvas and finds the combined rgb value for each pixel. The example then uses a slider to control the opacity of only the dark pixels (those with a combined RGB value of less than 150). You can drag the slider while the video is playing and the alpha value of the dark areas of each frame changes dynamically. The results are illustrated in Figure 17-3.

Figure 17-3  Image processing


Listing 17-3  Realtime image processing

<html>
<head>
    <title>Realtime Image Processing</title>
    <style>
        canvas {
            border-radius: 25px;
            border: 1px solid #404040;
        }
        .slider {
            position: absolute;
            top: 300px;
            left: 85px;
            width: 152px;
            height: 52px;
        }
        .bar {
            position: relative;
            top: 30px;
            width: 152px;
            height: 2px;
            background-color: #404040 ;
        }
        .knob {
            position: relative;
            left: 100px;
            border: 1px solid #404040;
            background-color: #C0C0C0 ;
            width: 50px;
            height: 50px;
            border-radius: 25px;
            text-align: center;
        }
    </style>
    <script type="text/javascript">
        var can, ctx, vid, play, vidTimer, mouseIsDown = 0,
            knobMid, pixData, pixels, knobX = 100;
 
        function init() {
            vid = document.getElementById("vid");
            play = document.getElementById("play");
            can = document.getElementById("can");
            ctx = can.getContext('2d');
            vid.addEventListener('ended', vidEnd, false);
            vid.addEventListener('play', vidSet, false);
            knobMid = knob.offsetWidth / 2;
            showVid();
        }
 
        function vidSet() {
            clearTimeout(vidTimer);
            vidTimer = setTimeout(showVid, 25);
        }
 
        function showVid() {
            ctx.drawImage(vid, 0, 0);
            processImage();
            if (!document.querySelector("video").paused)
            // Repeat 40 times a second to oversample 30 fps video
                vidTimer = setTimeout(showVid, 25);
        }
 
        function processImage() {
            // get pixel data for entire canvas
            pixels = ctx.getImageData(0, 0, can.width, can.height);
            pixData = pixels.data;
            // set alpha value using slider
            var alphaVal = parseInt(knobX * 2.55)
            // for each pixel
            for (i = 0; i < can.width * can.height; i++) {
                // get combined rgb value to determine brightness
                var rgbVal = pixData[i*4] + pixData[i*4 + 1] + pixData[i*4 + 2];
                // set alpha value for dark pixels to knob value
                if (rgbVal < 150)
                    pixData[i*4 + 3] = alphaVal;
            }
             // put modified data back into image object
             pixels.data = pixData;
             // blit modified image object to screen
             ctx.putImageData(pixels, 0, 0);
        }
 
 
        function playPauseVideo() {
            if (play.value == "Play") {
                vid.play();
                play.value = "Pause";
            }
            else {
                vid.pause();
                play.value = "Play";
            }
        }
 
        function vidEnd() {
                console.log(playing);
                play.value = "Play";
        }
 
        function mouseDown() {
            mouseIsDown = 1;
            mouseXY();
        }
 
        function mouseUp() {
            mouseIsDown = 0;
        }
 
        function mouseXY(e) {
            if (mouseIsDown) {
                if (!e)
                    var e = event;
                var mouseX = e.pageX - slider.offsetLeft;
                if (mouseX >= 0 && mouseX <= slider.offsetWidth) {
                    setKnob(mouseX);
                }
            }
        }
        function touchXY(e) {
            if (!e)
                var e = event;
            // slide, don't scroll
            e.preventDefault();
            var touchX = e.touches[0].pageX - slider.offsetLeft;
            if (touchX >= 0 && touchX <= slider.offsetWidth) {
                setKnob(touchX);
            }
        }
 
        function setKnob(x) {
            knobX = x - knobMid;
            knobX = Math.max(knobX, 0);
            knobX = Math.min(knobX, slider.offsetWidth - knob.offsetWidth);
            knob.style.left = knobX;
        }
    </script>
</head>
<body onload="init()" onmouseup="mouseUp()">
    <canvas id="can" height="270" width="480">
    </canvas>
    <p>
        <input type="button" value="Play" onclick="playPauseVideo()" id="play"
               style="font:18 pt Helvetica">
    </p>
    <video src="assets/myMovie.m4v" id="vid" style="display:none"></video>
    <div class="slider" id="slider"
         onmousedown="mouseDown()" onmousemove="mouseXY()"
         ontouchstart="touchXY()" ontouchmove="touchXY()">
        <div class="bar"></div>
        <div id="knob" class="knob">
            <br />
            Alpha
        </div>
    </div>
</body>
</html>


© 2012 Apple Inc. All Rights Reserved. (Last updated: 2012-09-19)

Did this document help you? Yes It's good, but... Not helpful...