/ JavaScript

Repo for the audio-driven image effects demo

I had a lengthy Slack discussion with a couple of friends about this Web Audio API demo the other day. It seems that for MacBook Pros, the audio wasn't picking up from the microphone at all, or in some cases only faintly.

Having the code baked right into the blog post makes it hard to talk specifics (which I covered a little here), so I pushed a repo and companion GitHub page where you can try it.

The flow from mic to image filter

In the original blog post demo, I was picking up the volume of the mic input, and putting a hard coded multiplier on that value. The value was then piped directly into the CSS image filters. Like this:

Mic input → (Vol value * Multiplier) → Filters

In the repo, I have added a UI slider to let you adjust the "mic sensitivity" yourself (what you are really adjusting is that multiplier).

A smarter move would be to have the multiplier auto-adjust to the ambient noise level based on the last N seconds of microphone input. I have an idea of how that should work, but at least for now, it's out of the scope of this tiny experiment.

Latency

I heard that in some cases, latency was an issue. You might notice this, for example, if you have music on and the snare drum pops, but you notice a little delay before the effects are triggered.

On my MacBook Air, I saw no problem with latency, but it's easy to imagine performance would be heavily affected by individual hardware setups.

There is a tradeoff to be made between latency and audio quality, or, put another way to describe this demo, reaction time of the effects versus how hard your computer needs to work to keep up. (If you've ever used a DAW to record music, you will be familiar with this phenomenon.)

In the original demo, I hard-coded the script processor's buffer at 2048 sample frames. This was the sweet spot on my MacBook Air, where the effects felt responsive but my fan stayed off.

I noticed while writing this post that MDN notes the following:

It is recommended for authors to not specify this buffer size and allow the implementation to pick a good buffer size to balance between latency and audio quality.

So in the repo, I pulled the 2048 value from the createScriptProcessor() call to let the system decide.

I can imagine cases where I might want to specify a buffer size, but in this experiment, the system is making good decisions on my MacBook Air.

(For thoughts on performance considerations when using script processors, see this fine post by Nick Thompson.)

No mic input

I do not know why MacBook Pros might not be picking up any audio at all. A few guesses do come to mind:

  1. MacBook Pro mics are less sensitive (if so, the UI slider I've added should help)
  2. MacBook Pros filter out their speaker output from the mic input to prevent feedback (maaaaaaaybe, but this flawlessly?)
  3. Web Audio API requires some polyfills that I haven't put into the repo (likely true, but that would be based on browser, not hardware)
  4. I'm coding it wrong (it's probably this)

Now go give the experiment a spin and check out the code if you are so inclined.