Creating a simple boomerang effect video in javascript

In the process of building on online video editor I wanted to scratch an itch that had been bugging me for a long time. Boomerang effect. It’s cheesy, fun and pretty useless. It’s also been said to be impossible to do on the web in real-time…. until now.
You can try the Instant Boomerang Creator if you don’t want to read the article. 🙂 (Note: I’ve only properly tested in Chrome as of today.

<p>In the process of building on online <a href="https://paul.kinlan.me/building-a-video-editor-on-the-web-with-the-web/">video
editor</a> I wanted to scratch
an itch that had been bugging me for a long time. Boomerang effect. It's cheesy,
fun and pretty useless. It's also been said to be impossible to do on the web in
real-time…. <a href="https://boomerang-video-instant.glitch.me/">until now</a>.</p>
<p>You can try the <a href="https://boomerang-video-instant.glitch.me/">Instant Boomerang
Creator</a> if you don't want to read
the article. 🙂 (Note: I've only properly tested in Chrome as of today.)</p>
<p>For the uninitiated, the boomerang effect is a video effect that is popular on
Instagram – a video will play forwards and then it will play backwards in an
infinte loop.</p>
<p>It turns out that it is not that easy to recreate on the web, especially if you
want it to happen instantly, but I would like to show you how I did it.</p>
<p>This article assumes that you know how to <a href="https://paul.kinlan.me/tags/mediarecorder/">record the frames from a video
stream using the MediaRecorder API</a>.</p>
<p>A simplistic approach would be to play a video forwards in a video element and
then when the playback position reaches the end of the video, set the
<code>playbackRate</code> to -1 and then repeat the process again when the playback
position reaches the beginning of the video. This, however has a number of
problems, the first that it's nearly impossible to get an accurate timestamp of
where the playback is in the video, and secondly the events that fire on the
<code><video></code> element aren't instant – both combined mean that the video playback
won't be smooth to the user, it also means that you will need to do the video
processing on the server because this would just be a simple video that is
recorded in normal forwards time.</p>
<p>An better solution would be to create a video file that had the forwards and
backwards frames in one video file and then just infinite loop that video. If we
can create the video we can remove all jank from the playback.</p>
<p>The solution I created uses a couple of little known API's <code>captureStream()</code>.
These API's are supported by Chromium based browsers and Firefox (there's a hint
it might be in Safari too).</p>
<p>The <code>captureStream()</code> API is available on <code><canvas></code>, <code><video></code> and <code><audio></code>
elements, and when called return a <code>MediaStream</code> object that can be used either
as a source for a <code><video></code> element, or as the input to the <code>MediaRecorder</code> API.</p>
<p>It is the <code>captureStream</code> method on the <code><canvas></code> element that is particularly
interesting. The method enables you to draw to the canvas (as you normally
would) and then at your own rate call <code>requestFrame()</code> on the video track of the
stream, which will push the contents of the <code><canvas></code> on to the MediaStream for
recording or for playback.</p>
<p>Armed with this method, my solution to get instant looping boomerang and a
recording of the single loop was as follows.</p>
<ol>
<li>Get the Web Cam stream via <code>getUserMedia</code></li>
<li>Create a 'frame buffer', to hold frames from the video camera stream when we
want to record</li>
<li>Create a canvas element that will act as our live video element</li>
<li>Set up a <code>requestAnimationFrame</code> and grab the frame from the camera stream
and pipe it to a hidden <code><video></code> element so that they can be drawn to a
visible <code><canvas></code>.</li>
<li>Set up a <code>MediaRecorder</code> whose input is the result of
<code>canvas.captureStream()</code></li>
<li>When the user wants to record
<ol>
<li>Set the <code>MediaRecorder</code> to record, then for each frame</li>
<li>Continue to draw each frame of video to the canvas (this is now the
forward part of the boomerang)</li>
<li>Add the <code>canvas</code> pixels to 'frame buffer'</li>
<li>When the user stops recording, clone and reverse the 'frame buffer'</li>
<li>Join the the two frame buffers together.</li>
<li>Stop rendering from the camera stream to the canvas, but use the frames
from the 'frame buffer', starting in the middle (the point the user
stopped recording)</li>
<li>As we progress through the frame buffer, stop the <code>MediaRecorder</code> when
at the last frame (the output will be exactly one forward loop and one
reverse loop).</li>
<li>Treat the frame buffer as a circular array, making an infinte loop of
boomerangtastic video.</li>
</ol>
</li>
</ol>
<p>The code is on my <a href="https://glitch.com/edit/#!/boomerang-video-instant?path=script.js:31:0">Glitch</a> but the core logic is below.</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-javascript" data-lang="javascript"><span style="color:#75715e">// There is some other set-up, but this will get the streams.
</span><span style="color:#75715e"></span><span style="color:#66d9ef">let</span> <span style="color:#a6e22e">canvasStream</span> <span style="color:#f92672">=</span> <span style="color:#a6e22e">canvasOutput</span>.<span style="color:#a6e22e">captureStream</span>(<span style="color:#ae81ff">0</span>);
<span style="color:#66d9ef">let</span> <span style="color:#a6e22e">canvasStreamTrack</span> <span style="color:#f92672">=</span> <span style="color:#a6e22e">canvasStream</span>.<span style="color:#a6e22e">getTracks</span>()[<span style="color:#ae81ff">0</span>];

<span style="color:#75715e">// The canvas stream will be played back on a video…
</span><span style="color:#75715e"></span><span style="color:#a6e22e">playbackVideo</span>.<span style="color:#a6e22e">srcObject</span> <span style="color:#f92672">=</span> <span style="color:#a6e22e">canvasStream</span>;
<span style="color:#75715e">// The canvas stream will be recorded
</span><span style="color:#75715e"></span><span style="color:#66d9ef">let</span> <span style="color:#a6e22e">rec</span> <span style="color:#f92672">=</span> <span style="color:#66d9ef">new</span> <span style="color:#a6e22e">MediaRecorder</span>(<span style="color:#a6e22e">canvasStream</span>, {<span style="color:#a6e22e">mimeType</span><span style="color:#f92672">:</span> <span style="color:#a6e22e">mimeType</span>});

<span style="color:#66d9ef">let</span> <span style="color:#a6e22e">renderFrame</span> <span style="color:#f92672">=</span> () => {
<span style="color:#75715e">// Every draw to the canvas will be rendered to the MediaStream
</span><span style="color:#75715e"></span> <span style="color:#75715e">// once requestFrame is called
</span><span style="color:#75715e"></span> <span style="color:#66d9ef">let</span> <span style="color:#a6e22e">imageData</span>;
<span style="color:#66d9ef">if</span> (<span style="color:#a6e22e">canvasStreamTrack</span> <span style="color:#f92672">===</span> <span style="color:#66d9ef">undefined</span>) <span style="color:#66d9ef">return</span>;

<span style="color:#66d9ef">if</span>(<span style="color:#a6e22e">recording</span>) {
<span style="color:#75715e">// The forwards part of the loop, draw to canvas anc
</span><span style="color:#75715e"></span> <span style="color:#75715e">// record the frame in to the 'frame buffer'
</span><span style="color:#75715e"></span> <span style="color:#a6e22e">context</span>.<span style="color:#a6e22e">drawImage</span>(<span style="color:#a6e22e">bufferVideo</span>, <span style="color:#ae81ff">0</span>, <span style="color:#ae81ff">0</span>);
<span style="color:#a6e22e">imageData</span> <span style="color:#f92672">=</span> <span style="color:#a6e22e">context</span>.<span style="color:#a6e22e">getImageData</span>(<span style="color:#ae81ff">0</span>, <span style="color:#ae81ff">0</span>, <span style="color:#a6e22e">width</span>, <span style="color:#a6e22e">height</span>);
<span style="color:#a6e22e">frames</span>.<span style="color:#a6e22e">push</span>(<span style="color:#a6e22e">imageData</span>);
}
<span style="color:#66d9ef">else</span> {
<span style="color:#66d9ef">if</span> (<span style="color:#a6e22e">playback</span> <span style="color:#f92672">===</span> <span style="color:#66d9ef">false</span>) {
<span style="color:#75715e">// Use the camera stream straight to canvas
</span><span style="color:#75715e"></span> <span style="color:#a6e22e">context</span>.<span style="color:#a6e22e">drawImage</span>(<span style="color:#a6e22e">bufferVideo</span>, <span style="color:#ae81ff">0</span>, <span style="color:#ae81ff">0</span>);
}
<span style="color:#66d9ef">else</span> {
<span style="color:#a6e22e">canvasOutput</span>.<span style="color:#a6e22e">width</span> <span style="color:#f92672">=</span> <span style="color:#a6e22e">canvasOutput</span>.<span style="color:#a6e22e">width</span>;
<span style="color:#75715e">// Loop through the frame buffer and draw to canvas
</span><span style="color:#75715e"></span> <span style="color:#a6e22e">frameIdx</span> <span style="color:#f92672">=</span> (<span style="color:#f92672">++</span><span style="color:#a6e22e">frameIdx</span>) <span style="color:#f92672">%</span> <span style="color:#a6e22e">frames</span>.<span style="color:#a6e22e">length</span>;
<span style="color:#a6e22e">context</span>.<span style="color:#a6e22e">putImageData</span>(<span style="color:#a6e22e">frames</span>[<span style="color:#a6e22e">frameIdx</span>], <span style="color:#ae81ff">0</span>, <span style="color:#ae81ff">0</span>);
<span style="color:#66d9ef">if</span> (<span style="color:#a6e22e">frameIdx</span> <span style="color:#f92672">===</span> <span style="color:#ae81ff">0</span> <span style="color:#f92672">&&</span> <span style="color:#a6e22e">rec</span>.<span style="color:#a6e22e">state</span> <span style="color:#f92672">!=</span> <span style="color:#e6db74">'inactive'</span>) {
<span style="color:#75715e">// We are at the last frame of the reversal, stop recording
</span><span style="color:#75715e"></span> <span style="color:#a6e22e">rec</span>.<span style="color:#a6e22e">stop</span>();
}
}
}

<span style="color:#75715e">// Make sure the updates to canvas are rendered on to the stream.
</span><span style="color:#75715e"></span> <span style="color:#a6e22e">canvasStreamTrack</span>.<span style="color:#a6e22e">requestFrame</span>();
<span style="color:#a6e22e">requestAnimationFrame</span>(<span style="color:#a6e22e">renderFrame</span>);
};

<span style="color:#66d9ef">let</span> <span style="color:#a6e22e">startRecording</span> <span style="color:#f92672">=</span> <span style="color:#a6e22e">async</span> () => {
<span style="color:#a6e22e">recording</span> <span style="color:#f92672">=</span> <span style="color:#66d9ef">true</span>;
<span style="color:#a6e22e">frames</span> <span style="color:#f92672">=</span> [];

<span style="color:#a6e22e">rec</span>.<span style="color:#a6e22e">start</span>(<span style="color:#ae81ff">10</span>);
}

<span style="color:#66d9ef">let</span> <span style="color:#a6e22e">stopRecording</span> <span style="color:#f92672">=</span> () => {
<span style="color:#a6e22e">recording</span> <span style="color:#f92672">=</span> <span style="color:#66d9ef">false</span>;
<span style="color:#a6e22e">playback</span> <span style="color:#f92672">=</span> <span style="color:#66d9ef">true</span>;

<span style="color:#75715e">// Take the frames, and reverse
</span><span style="color:#75715e"></span> <span style="color:#a6e22e">frames</span> <span style="color:#f92672">=</span> […<span style="color:#a6e22e">frames</span>, …<span style="color:#a6e22e">frames</span>.<span style="color:#a6e22e">slice</span>(<span style="color:#ae81ff">0</span>).<span style="color:#a6e22e">reverse</span>()];
<span style="color:#75715e">// Set the start of the playback to be the first reverse frame
</span><span style="color:#75715e"></span> <span style="color:#a6e22e">frameIdx</span> <span style="color:#f92672">=</span> Math.<span style="color:#a6e22e">round</span>(<span style="color:#a6e22e">frames</span>.<span style="color:#a6e22e">length</span> <span style="color:#f92672">/</span> <span style="color:#ae81ff">2</span>);
}
</code></pre></div><p>And that's it. Instant Live-Boomerang, with a saved recording.</p>


Print Share Comment Cite Upload Translate
APA
Paul Kinlan | Sciencx (2024-03-29T13:33:16+00:00) » Creating a simple boomerang effect video in javascript. Retrieved from https://www.scien.cx/2018/11/05/creating-a-simple-boomerang-effect-video-in-javascript/.
MLA
" » Creating a simple boomerang effect video in javascript." Paul Kinlan | Sciencx - Monday November 5, 2018, https://www.scien.cx/2018/11/05/creating-a-simple-boomerang-effect-video-in-javascript/
HARVARD
Paul Kinlan | Sciencx Monday November 5, 2018 » Creating a simple boomerang effect video in javascript., viewed 2024-03-29T13:33:16+00:00,<https://www.scien.cx/2018/11/05/creating-a-simple-boomerang-effect-video-in-javascript/>
VANCOUVER
Paul Kinlan | Sciencx - » Creating a simple boomerang effect video in javascript. [Internet]. [Accessed 2024-03-29T13:33:16+00:00]. Available from: https://www.scien.cx/2018/11/05/creating-a-simple-boomerang-effect-video-in-javascript/
CHICAGO
" » Creating a simple boomerang effect video in javascript." Paul Kinlan | Sciencx - Accessed 2024-03-29T13:33:16+00:00. https://www.scien.cx/2018/11/05/creating-a-simple-boomerang-effect-video-in-javascript/
IEEE
" » Creating a simple boomerang effect video in javascript." Paul Kinlan | Sciencx [Online]. Available: https://www.scien.cx/2018/11/05/creating-a-simple-boomerang-effect-video-in-javascript/. [Accessed: 2024-03-29T13:33:16+00:00]
rf:citation
» Creating a simple boomerang effect video in javascript | Paul Kinlan | Sciencx | https://www.scien.cx/2018/11/05/creating-a-simple-boomerang-effect-video-in-javascript/ | 2024-03-29T13:33:16+00:00
https://github.com/addpipe/simple-recorderjs-demo