HTML5 Rocks

HTML5 Rocks

Web Audio live audio input - now on Android!

By Chris Wilson at

One of the consistent questions I keep fielding over on Stack Overflow is “why doesn’t audio input work?” – to which the answer kept turning out to be “because you’re testing on Android, and we don’t have it hooked up yet.”

Well, I’m happy to announce that the Beta version of Chrome for Android (v31.0.1650+) has support for audio input in Web Audio! Check out my Audio Recorder demo on an Android device with Chrome Beta. We’re still testing it out with all devices; I’ve personally tested on a Nexus 4, a Galaxy S4 and a Nexus 7, but if you run into issues on other devices, please file them.

When I saw the support get checked in, I flipped back through some of the audio input demos I’ve done in the past to find a good demo to show it off. I quickly found that my Audio Recorder demo functioned well on mobile, but it wasn’t really designed for a good user experience on mobile devices.

So, I quickly used the skills I’m teaching in the upcoming Mobile Web Development course to whip it into shape – viewport, media queries, and flexbox to the rescue! Be sure to preregister for the course if you’re interested in taking your web development skills to the mobile world, too!

Live Web Audio Input Enabled!

By Chris Wilson at

I'm really excited by a new feature that went in to yesterday's Chrome Canary build (23.0.1270.0) - the ability to get low-latency access to live audio from a microphone or other audio input on OSX! (This has not yet been enabled on Windows - but don't worry, we're working on it!)

[UPDATE Oct 8, 2012: live audio input is now enabled for Windows, as long as the input and output device are using the same sample rate!]

To enable this, you need to go into chrome://flags/ and enable the "Web Audio Input" item near the bottom, and relaunch the browser; now you're ready to roll! Note: If you're using a microphone, you may need to use headphones for any output in order to avoid feedback. If you are using a different audio source, such as a guitar or external audio feed, or there's no audio output from the demo, this may not be a problem. You can test out live audio input by checking out the spectrum of your input using the live input visualizer.

For those Web Audio coders among you, here's how to request the audio input stream, and get a node to connect to any processing graph you like!

// success callback when requesting audio input stream
function gotStream(stream) {
    window.AudioContext = window.AudioContext || window.webkitAudioContext;
    var audioContext = new AudioContext();

    // Create an AudioNode from the stream.
    var mediaStreamSource = audioContext.createMediaStreamSource( stream );

    // Connect it to the destination to hear yourself (or any other node for processing!)
    mediaStreamSource.connect( audioContext.destination );
}

navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia;
navigator.getUserMedia( {audio:true}, gotStream );

There are many rich possibilities for low-latency audio input, particularly in the musical space. You can see a quick example of how to make use of this in a simple pitch detector I threw together - try plugging in a guitar, or even just whistling into the microphone.

And, as promised, I've added live audio as an input source to the Vocoder I wrote for Google IO - just select "live input" under modulator. You may need to adjust the Modulator Gain and the Synth Level. There's a slight lag due to processing (not due to input latency). Now that I have live audio input, it's time for another round of tweaking!

Finally, you may want to take a look at the collection of my web audio demos - by the time you read this, I may have some more live audio demos up!

WebGL and Web Audio API demo roundup

By Ilmari Heikkinen at

Here’s a look at some cool WebGL and Web Audio API demos that I’ve seen over the past couple weeks.

EVE Online ship viewer is a great-looking online ship viewer app built with WebGL. Very nice way to showcase the artwork in the game universe.

Web Audio API samples page has several compelling examples on how to do audio processing using it. WebGL City is one of the demos linked from the samples page. It’s a small demo of a helicopter flying around a night cityscape. The helicopter (disable music by pressing 'm’, enable helicopter sound by pressing 'n’) uses Web Audio APIs spatial audio features to pan the helicopter audio from one speaker to the other.

Some enterprising soul implemented a snake game using nothing but a WebGL fragment shader on the GLSL Sandbox. I’m flabbergasted.

The Big Bang may look like any other WebGL particle animation, but the particle simulation is actually run on the GPU. The simulator is a fragment shader that reads the previous particle positions from a texture and writes the new particle positions into a FBO texture.

Blocky Earth takes Google Earth data and MineCrafts it. It communicates differences in height well. For example, I was looking at Australia and the Antarctic ice sheet, and you can see how the continental ice is several kilometers thick.

The Midem Music Machine is a fun music demo by Mr.doob and Paul Lamere. It’s a sort of a ball-driven music box with balls bouncing off bits 'n’ bops. CreativeJS has a good writeup on it, check it out.

Continuing on the computer music visualization theme, I recently ran across this page about bytebeat, a form of music generated by minimalistic code formula. The page links to one cool WebGL visualization of the music. Gregg Tavares took to the idea and built a bytebeat sandbox for making and sharing your own bytebeat tunes directly from the browser.

HTML5 <audio> and the Web Audio API are BFFs!

By Eric Bidelman at

DEMO

As part of the MediaStream Integration with WebRTC, the Web Audio API recently landed an undercover gem known as createMediaElementSource(). Basically, it allows you to hook up an HTML5 <audio> element as the input source to the API. In layman's terms...you can visualize HTML5 audio, do realtime sound mutations, filtering, etc!

Normally, the Web Audio API works by loading a song via XHR2, file input, whatever,....and you're off. Instead, this hook allows you to combine HTML5 <audio> with the visualization, filter, and processing power of the Web Audio API.

Integrating with<audio> is ideal for streaming fairly long audio assets. Say your file is 2 hours long. You don't want to decode that entire thing! It's also interesting if you want to build a high-level media player API (and UI) for play/pause/seek, but wish to apply some additional processing/analysis.

Here's what it looks like:

// Create an <audio> element dynamically.
var audio = new Audio();
audio.src = 'myfile.mp3';
audio.controls = true;
audio.autoplay = true;
document.body.appendChild(audio);

var context = new webkitAudioContext();
var analyser = context.createAnalyser();

// Wait for window.onload to fire. See crbug.com/112368
window.addEventListener('load', function(e) {
  // Our <audio> element will be the audio source.
  var source = context.createMediaElementSource(audio);
  source.connect(analyser);
  analyser.connect(context.destination);

  // ...call requestAnimationFrame() and render the analyser's output to canvas.
}, false);

As noted in the code, there's a bug that requires the source setup to happen after window.onload.

The next logical step is to fix crbub.com/112367. Once that puppy is ready, you'll be able to wire up WebRTC (the navigator.getUserMedia() API in particular) to pipe audio input (e.g mic, mixer, guitar) to an <audio> tag, then visualize it using the Web Audio API. Mega boom!

Web Audio FAQ

By Boris Smus at

Over the past few months, the WebKit Web Audio API has emerged as a compelling platform for games and audio applications on the web. As developers familiarize themselves with it, I hear similar questions creep up repeatedly. This quick update is an attempt to address some of the more frequently asked questions to make your experience with the Web Audio API more pleasant.

Q: Halp, I can't make sounds!

A: If you're new to the Web Audio API, take a look at the getting started tutorial, or Eric's recipe for playing audio based on user interaction.

Q. How many Audio Contexts should I have?

A: Generally, you should include one AudioContext per page, and a single audio context can support many nodes connected to it. Though you may include multiple AudioContexts on a single page, this can lead to a performance hit.

Q: I’ve got an AudioBufferSourceNode, that I just played back with noteOn(), and I want to play it again, but noteOn() doesn’t do anything! Help!

A: Once a source node has finished playing back, it can’t play back more. To play back the underlying buffer again, you should create a new AudioBufferSourceNode and call noteOn().

Though re-creating the source node may feel inefficient, source nodes are heavily optimized for this pattern. Plus, if you keep a handle to the AudioBuffer, you don't need to make another request to the asset to play the same sound again. If you find yourself needing to repeat this pattern, encapsulate playback with a simple helper function like playSound(buffer).

Q: When playing back a sound, why do you need to make a new source node every time?

A: The idea of this architecture is to decouple audio asset from playback state. Taking a record player analogy, buffers are analogous to records and sources to play-heads. Because many applications involve multiple versions of the same buffer playing simultaneously, this pattern is essential.

Q: How can I process sound from audio and video tags?

A: MediaElementAudioSourceNode is in the works! When available, it will work roughly as follows (adding a filter effect to a sample playing via the audio tag):

<audio src="sounds/sample.wav" controls>

var audioElement = document.querySelector('audio');
var mediaSourceNode = context.createMediaElementSource(audioElement);
mediaSourceNode.connect(filter);
filter.connect(context.destination);

This feature is tracked in this crbug. Note that in this setup, there is no need to call mediaSourceNode.noteOn(), the audio tag controls playback.

Q: When can I get sound from a Microphone?

A: The audio input part of this will be implemented as part of WebRTC using getUserMedia, and be available as a special source node in the Web Audio API. It will work in conjunction with createMediaElementSource.

Q: How can I check when an AudioSourceNode has finished playing?

A: Currently you have to use a JavaScript timer since Web Audio API does not support this functionality. The following snippet from the Getting Started with Web Audio API tutorial is an example of this in action:

// Assume source and buffer are previously defined.
source.noteOn(0);
var timer = setTimeout(function() {
  console.log('playback finished');
}, buffer.duration * 1000);

There is an open bug to make Web Audio API implement a more accurate callback.

Q: Loading sounds causes the whole UI thread to lock up and my UI becomes unresponsive. Help!

A: Use the decodeAudioData API for asynchronous loading to avoid blocking the main thread. See this example.

Q: Can the Web Audio API be used to process sounds faster than realtime?

A: Yes, a solution is being worked on. Please stay tuned!

Q: I’ve made an awesome Web Audio API application, but whenever the tab it's running in goes in the background, sounds go all weird!

A: This is probably because you are using setTimeouts, which behave differently if the page is backgrounded. In the future the Web Audio API will be able to callback at specific times using the web audio’s internal timer (context.currentTime attribute). For more information, please see this feature request.

In general, it may be a good idea to stop playback when your app goes into the background. You can detect when a page goes to the background using the Page Visibility API.

Q: How can I change the pitch of a sound using the Web Audio API?

A: Change the playbackRate on the source node.

Q: Can I change pitch without changing speed?

A: The Web Audio API could have a PitchNode in the audio context, but this is hard to implement. This is because there is no straightforward pitch shifting algorithm in audio community. Known techniques create artifacts, especially in cases where the pitch shift is large. There are two kinds of approaches to tackle this problem:

  • Time domain algorithms, which cause repeated segment echoes artifacts.
  • Frequency domain techniques, which cause reverberant sound artifacts.

Though there is no native node for doing these techniques, you can do it in with a JavaScriptAudioNode. This code snippet might serve as a starting point.

Q: How can I create an AudioContext at a sample rate of my choosing?

A: Currently there is no support for this, but we’re looking into it. See this feature request.

If you have additional questions, feel free to ask them on StackOverflow using the web-audio tag.

Multiplayer Audio Fun

By Ernest Delgado at

Using the Web Audio API, WebSockets and a very nice designed UI here comes a demo where you can generante notes on the fly and with other people in real time.
We will keep posting quick updates with the demos that developers around the world make with HTML5.

Contra in HTML5 + Web Audio API

By Eric Bidelman at

Thanks to the power of GWT, HTML5, and the Web Audio API we can build the originator of everyone's favorite cheat code, Contra: http://nes-sound.brad-rydzewski.appspot.com/

Check out the open source NES Emulator in HTML5, gwt-nes-port.