Start learning
Part 3/Graphics and Media

Web Audio & WebCodecs

An <audio> element plays a file. That is the whole extent of its ambition — hand it a URL, get sound. The moment you want to make a tone from nothing, fade one signal into another on a musical beat, or read the raw spectrum of what is playing to drive a visualizer, you have left its world entirely.

Two lower-level APIs pick up where the media elements stop. Web Audio builds sound out of small connected processing blocks — a modular synth you wire up in JavaScript. WebCodecs hands you the browser’s own hardware video and audio codecs as plain objects, so you can encode and decode frames yourself instead of shipping them straight to a <video> tag. One is old and boringly reliable; the other is new and worth understanding before you reach for it. This article covers both.

The mental model: a graph, not a stream

The single idea that makes Web Audio click is this: you do not write audio, you wire it. Sound flows through a graph of nodes. Each node does one job — produce a tone, change the volume, filter frequencies, measure the signal — and you connect its output to the input of the next. Signal enters at a source, passes through however many effect nodes you like, and finally reaches the destination (your speakers).

A Web Audio graph. Sound flows left to right from a source, through effect nodes, to the destination. Each connect() call draws one of these arrows. The moving dot is the signal.

This graph is declarative and live. Once wired, it runs on a dedicated audio thread at the sample rate (typically 48000 samples per second), completely independent of your JavaScript. You are not pushing samples in a loop; you describe the topology once and adjust parameters over time. That separation is why Web Audio can stay glitch-free even while your main thread is busy.

Booting an AudioContext

Everything hangs off an AudioContext. It owns the graph, the audio clock, and the connection to the hardware.

const ctx = new AudioContext();
console.log(ctx.sampleRate);   // e.g. 48000
console.log(ctx.currentTime);  // seconds since the context started, a high-precision clock

There is one rule you cannot skip: a page cannot start audio without a user gesture. Browsers block autoplaying sound, so a freshly created context often begins in a "suspended" state. It will not produce sound until you resume() it, and that call only succeeds from inside a real user interaction — a click, a keypress, a tap.

const ctx = new AudioContext();

playButton.addEventListener('click', async () => {
  if (ctx.state === 'suspended') {
    await ctx.resume();   // must originate from a gesture
  }
  startTone();
});

Your first tone: oscillator → gain → speakers

The smallest useful graph is a source, a volume control, and the destination. An OscillatorNode generates a periodic waveform — a pure mathematical tone. A GainNode scales its amplitude (volume).

const ctx = new AudioContext();

const osc = new OscillatorNode(ctx, { type: 'sine', frequency: 440 }); // A4
const gain = new GainNode(ctx, { gain: 0.2 });                          // 20% volume

osc.connect(gain).connect(ctx.destination);

osc.start();               // begin producing sound now
osc.stop(ctx.currentTime + 1); // schedule it to stop 1 second from now

Four things are worth pausing on. The type can be 'sine', 'square', 'sawtooth', or 'triangle' — each a different timbre. connect() returns the node you connected to, which is why the calls chain. A source node is single-use: once you stop() an oscillator you cannot restart it; you build a new one. And start() / stop() take an absolute time on the audio clock, not a delay — passing nothing means “right now.”

Why 0.2 and not 1.0

Set that gain to 1.0 and stack a few oscillators and you will clip — the summed waveform exceeds the range the hardware can represent, and clipping sounds like harsh digital crackle. Keep individual sources low and leave headroom. Gain is a linear multiplier: 0.5 is half the amplitude, roughly 6 dB quieter.

Wire up the graph yourself below. The Play button creates the AudioContext on click (satisfying the gesture rule), then builds an OscillatorNode → GainNode → destination chain. Switch the waveform to hear how the same pitch changes timbre, and drag the slider to retune the live oscillator through its frequency param.

interactiveOscillator → gain → speakers

AudioParam: values that change over time

The frequency on an oscillator and the gain on a gain node are not plain numbers. They are AudioParam objects, and that distinction is the heart of precise audio. An AudioParam can hold a value and a schedule of future changes, all evaluated on the audio thread with sample accuracy.

Setting .value directly works but jumps instantly, which produces an audible click. Instead you schedule ramps against ctx.currentTime:

const now = ctx.currentTime;
const gain = new GainNode(ctx, { gain: 0 });

// An amplitude envelope: silence → quick attack → slow decay
gain.gain.setValueAtTime(0, now);
gain.gain.linearRampToValueAtTime(0.3, now + 0.02);  // 20ms attack
gain.gain.exponentialRampToValueAtTime(0.0001, now + 1.5); // decay tail
time → ctx.currentTime + tattackexponential decay
A gain envelope scheduled on an AudioParam. The value ramps up fast (attack), then decays smoothly. Each scheduling call sets one segment; the audio thread interpolates between the control points.

Hear the difference an envelope makes. Each Pluck builds a fresh oscillator and gain, then schedules a linear attack up to full volume followed by an exponential decay to near-silence. Shorten the attack for a percussive snap; lengthen the decay for a lingering bell. A bare tone with no envelope clicks on and off; a shaped one breathes.

interactiveScheduling an amplitude envelope

Playing recorded audio: AudioBufferSourceNode

Oscillators make synthetic tones. To play a sample — a drum hit, a sound effect, a music clip — you decode a file into an AudioBuffer and feed it through an AudioBufferSourceNode.

const res = await fetch('/sfx/snare.wav');
const bytes = await res.arrayBuffer();
const buffer = await ctx.decodeAudioData(bytes); // decode to raw samples

function hit() {
  const src = new AudioBufferSourceNode(ctx, { buffer });
  src.connect(ctx.destination);
  src.start();
}

decodeAudioData runs once; you keep the resulting AudioBuffer and spin up a fresh, cheap AudioBufferSourceNode every time you want to trigger it. Like oscillators, buffer sources are single-use — this is by design, so retriggering a sound never fights over playback position. You can also set playbackRate to pitch-shift, or loop: true for sustained beds.

Scheduling: the audio clock is not your clock

Here is the mistake everyone makes first: sequencing sounds with setTimeout. The main thread is jittery — a garbage-collection pause or a heavy render will shove your callback tens of milliseconds late, and a beat that drifts is instantly noticeable. The fix is to schedule against ctx.currentTime, the audio thread’s own high-resolution clock, ahead of time.

nowlook-ahead window (~100ms)horizonbookedbookedbookedwaiting
Look-ahead scheduling. A timer that wakes every ~25ms peeks a short window into the future and books any notes due before the horizon. The audio thread plays them at their exact sample, immune to main-thread jitter.
let nextNoteTime = ctx.currentTime;
const tempo = 120;                 // BPM
const secondsPerBeat = 60 / tempo;
const lookAhead = 0.1;             // schedule 100ms into the future

function scheduler() {
  // Book every note that falls before the horizon
  while (nextNoteTime < ctx.currentTime + lookAhead) {
    playNoteAt(nextNoteTime);
    nextNoteTime += secondsPerBeat;
  }
}
setInterval(scheduler, 25);        // wake often, but do little

function playNoteAt(time) {
  const osc = new OscillatorNode(ctx, { frequency: 440 });
  const g = new GainNode(ctx, { gain: 0 });
  osc.connect(g).connect(ctx.destination);
  g.gain.setValueAtTime(0.3, time);
  g.gain.exponentialRampToValueAtTime(0.001, time + 0.2);
  osc.start(time);
  osc.stop(time + 0.2);
}

The setInterval is deliberately imprecise — it only decides what to schedule. The time argument passed to start() is what actually lands the note dead on the beat. This look-ahead pattern is the backbone of every browser metronome, sequencer, and rhythm game.

Shaping tone with a filter

A BiquadFilterNode is a second-order filter you drop into the chain to carve frequencies. Set its type and it becomes a lowpass, highpass, bandpass, notch, or shelf. A lowpass with a swept cutoff is the classic “wah” or synth sweep.

const filter = new BiquadFilterNode(ctx, {
  type: 'lowpass',
  frequency: 800,   // cutoff in Hz
  Q: 8,             // resonance — a peak at the cutoff
});

osc.connect(filter).connect(ctx.destination);

// Sweep the cutoff from 200Hz up to 4kHz over a second
filter.frequency.setValueAtTime(200, ctx.currentTime);
filter.frequency.exponentialRampToValueAtTime(4000, ctx.currentTime + 1);

Because frequency and Q are AudioParams, the same ramp machinery from earlier animates them. Nothing new to learn — the graph model reuses one idea everywhere.

Reading the signal: AnalyserNode

An AnalyserNode is a tap. It passes audio through unchanged but lets you read the current signal on demand — either as a waveform (time domain) or as a spectrum (frequency domain, via an FFT). This is how you build meters, oscilloscopes, and bar visualizers.

const analyser = new AnalyserNode(ctx, { fftSize: 256 });
source.connect(analyser);
analyser.connect(ctx.destination); // pass-through, so you still hear it

// frequencyBinCount is always fftSize / 2
const bins = new Uint8Array(analyser.frequencyBinCount); // 128 values, 0–255

function draw() {
  requestAnimationFrame(draw);
  analyser.getByteFrequencyData(bins);   // fills `bins` with the current spectrum
  for (let i = 0; i < bins.length; i++) {
    const height = bins[i]; // 0–255, low index = low frequency
    // ...paint a bar `height` tall...
  }
}
draw();
low Hzhigh Hz
The AnalyserNode fills a Uint8Array with the current spectrum each frame — 0 to 255 per frequency bin, low frequencies on the left. A render loop reads it and draws bars. The analyser does not alter the sound.

fftSize must be a power of two (32 up to 32768). Bigger means finer frequency resolution but coarser time resolution and more work — 256 or 512 is plenty for a bar display. For a waveform scope, call getByteTimeDomainData instead. Both write into an array you own, so allocate the Uint8Array once outside the loop, not every frame.

Here is the whole chapter in one graph. This demo generates white noise, pushes it through a lowpass BiquadFilterNode whose cutoff is swept by a slow oscillator (an LFO wired straight into the filter’s frequency param), then taps the result with an AnalyserNode before it reaches the speakers. A requestAnimationFrame loop reads getByteFrequencyData every frame and paints the bars — watch them slide as the moving cutoff lets more or fewer high frequencies through.

interactiveLive spectrum from an AnalyserNode

Where Web Audio stands

The API is a W3C recommendation and has been Baseline widely available since 2021 — you can use AudioContext, oscillators, gains, filters, and analysers everywhere without a second thought, on desktop and mobile. The only universal caveat is the user-gesture requirement for starting sound, which is policy, not support. Prefixed webkitAudioContext is ancient history; you can drop it.

WebCodecs: the codecs, unbundled

Now change problems. You are not synthesizing sound — you have encoded media and you want to do something the <video> element will not do for you: transcode a file, process every frame before display, or push ultra-low-latency frames over a data channel. That means getting between the compressed bytes and the raw pixels, and that is exactly the seam WebCodecs exposes.

Every browser ships highly optimized, often hardware-accelerated codecs — that is how a <video> plays H.264 or AV1 smoothly. Historically those codecs were sealed inside the media pipeline. WebCodecs unseals them and hands you four low-level classes: VideoEncoder, VideoDecoder, AudioEncoder, AudioDecoder. You feed raw data in one side and get compressed chunks out the other, or vice versa.

encodeVideoFrameraw pixelsVideoEncoderEncodedVideoChunk.encode()output()decodeEncodedVideoChunkVideoDecoderVideoFrameraw pixels.decode()output()
The WebCodecs pipeline is symmetric. An encoder turns raw VideoFrames into compressed EncodedVideoChunks; a decoder does the reverse. You supply container parsing/muxing and transport yourself — WebCodecs only does the codec step.

The two units of currency

WebCodecs deals in two kinds of object, and understanding the split is most of the battle:

  • VideoFrame — one uncompressed frame. It wraps actual pixel memory (often on the GPU), carries a timestamp in microseconds, and is expensive. You can construct one from a <canvas>, an ImageBitmap, a <video>, or a raw buffer.
  • EncodedVideoChunk — one compressed frame: an opaque blob of codec bytes plus a type of "key" or "delta" and a timestamp. Small and cheap.

Encoders eat frames and emit chunks. Decoders eat chunks and emit frames. Same story for AudioData and EncodedAudioChunk on the audio side.

Encoding frames

An encoder is built with two callbacks — where output goes, and where errors go — then configured with a codec and dimensions.

const chunks = [];

const encoder = new VideoEncoder({
  output(chunk, meta) {
    // chunk is an EncodedVideoChunk. `meta` may carry decoder config for keyframes.
    chunks.push(chunk);
  },
  error(e) { console.error(e); },
});

encoder.configure({
  codec: 'vp09.00.10.08',  // fully-qualified codec string, not just "vp9"
  width: 1280,
  height: 720,
  bitrate: 2_000_000,      // 2 Mbps
  framerate: 30,
});

// Feed frames — e.g. drawn on a canvas
for (let i = 0; i < frameCount; i++) {
  const frame = new VideoFrame(canvas, { timestamp: (i * 1e6) / 30 });
  encoder.encode(frame, { keyFrame: i % 60 === 0 });
  frame.close();           // release GPU memory immediately — critical
}

await encoder.flush();     // wait for all pending work to drain

Two rules will save you hours. First, the codec string must be fully specified — 'vp09.00.10.08', 'avc1.42001f', 'av01.0.04M.08' — because it pins profile, level, and bit depth. A bare 'vp9' throws. Call VideoEncoder.isConfigSupported(config) first and check .supported; hardware support varies by machine.

Decoding chunks

Decoding is the mirror image. Configure with the codec (and, for many formats, a description of extradata pulled from the container), feed chunks, receive frames.

const decoder = new VideoDecoder({
  output(frame) {
    ctx2d.drawImage(frame, 0, 0);   // paint it
    frame.close();                  // then release it
  },
  error(e) { console.error(e); },
});

decoder.configure({ codec: 'vp09.00.10.08' });

for (const chunk of chunks) {
  decoder.decode(chunk);
}
await decoder.flush();

Decoding is stateful: a "delta" chunk only makes sense relative to the keyframe before it, so you must feed chunks in order and start from a keyframe. Feed a delta with no preceding key and you get garbage or an error.

Backpressure is your job

Both encoder and decoder expose encodeQueueSize / decodeQueueSize — how many operations are still pending. WebCodecs will happily accept work faster than the hardware can process it, and if you never look, you build an unbounded backlog and blow through memory. Real pipelines watch the queue and pause the source (or the ramp of requestVideoFrameCallback) when it climbs.

What WebCodecs is not

WebCodecs is deliberately narrow. It does only the codec step. It does not parse or produce container files (MP4, WebM), it does not handle transport, and it does not do audio/video sync for you. That is a feature — it composes with the other pieces.

<video> / <audio>Play a stream or file. Zero setup, zero frame access.
MediaRecorderRecord a MediaStream to a finished blob. Simple, but you cannot touch individual frames or tune latency.
WebCodecsRaw frames in and out. Per-frame processing, custom muxing, sub-frame latency. You assemble the rest of the pipeline.
Where WebCodecs sits versus the higher-level media APIs. MediaRecorder gives you a finished file with no control; a plain video element gives you playback with no access. WebCodecs trades convenience for direct frame-level control.

A realistic transcoder looks like: a demuxer (often WebAssembly, e.g. an MP4 parser) pulls EncodedVideoChunks from the file → VideoDecoder → your processing on each VideoFrame → VideoEncoder → a muxer writes a new container. WebCodecs is the two engine blocks in the middle; you bring the plumbing.

Good fits

  • Low-latency streaming — encode frames and ship them over WebTransport or an RTCDataChannel yourself, skipping the buffering a <video> pipeline imposes. Cloud gaming and remote-desktop tools live here.
  • Client-side transcoding / editing — decode, apply effects or trims frame by frame, re-encode, all without a server round trip.
  • Frame extraction — pull exact frames for thumbnails, ML inference, or scrubbing.

Where WebCodecs stands (July 2026)

This is the part to state plainly, because WebCodecs is much newer than Web Audio and support is uneven:

  • Chrome / Edge: full support since v94 (2021) on desktop; Chrome for Android caught up much later (Chrome 147).
  • Safari: full support landed in Safari 26.0 across macOS, iOS, and iPadOS. Versions 16.4 through 18.7 shipped only the video interfaces — no audio codec classes — so on slightly older Safari you get VideoDecoder but not AudioDecoder.
  • Firefox: enabled on desktop from Firefox 130; on Firefox for Android, VideoDecoder is still absent, so you need a fallback path there.

Run the exact probe against your browser. It checks each WebCodecs class with the in operator, then — if VideoEncoder exists — asks isConfigSupported whether this machine can actually encode a couple of common codecs. The answer depends on your OS and hardware, so a friend’s result may differ from yours.

interactiveFeature-detecting WebCodecs

It is a W3C Working Draft, not yet Baseline. For production, that means: gate features behind detection, test on your real target browsers, and be ready with a MediaRecorder or server-side path where WebCodecs is missing.

Two APIs, one lesson

Both APIs reward the same instinct — stop treating media as an opaque blob you hand to an element, and start treating it as data you route and transform. Web Audio gives you a live graph of processing nodes for sound. WebCodecs gives you the raw codec engines for video and audio. One is universally available and safe to lean on today; the other is powerful, newer, and demands feature detection. Reach for them when the built-in elements run out of road.

Summary

  • Web Audio models sound as a graph of nodes: sources → effects → ctx.destination, wired with connect() and running on a dedicated audio thread.
  • An AudioContext starts suspended; you must resume() it from a real user gesture before any sound plays.
  • Parameters like frequency and gain are AudioParams — schedule ramps (setValueAtTime, linearRampToValueAtTime, exponentialRampToValueAtTime) against ctx.currentTime instead of setting .value to avoid clicks. Exponential ramps cannot target zero.
  • Sequence with look-ahead scheduling: a coarse timer books notes a short window into the future, and the exact time passed to start() keeps them sample-accurate despite main-thread jitter.
  • AnalyserNode taps the signal without altering it; getByteFrequencyData fills a Uint8Array (0–255) you allocate once and read every frame for visualizers.
  • Web Audio is Baseline widely available (since 2021) — safe everywhere.
  • WebCodecs exposes the browser’s built-in codecs as VideoEncoder / VideoDecoder / AudioEncoder / AudioDecoder, trading raw VideoFrames for compressed EncodedVideoChunks and back.
  • Use fully-qualified codec strings, close() every frame immediately, watch the queue sizes for backpressure, and prefer a Worker for sustained pipelines.
  • WebCodecs is newer and uneven (full Safari support only from 26.0, Firefox desktop-only, mobile gaps) — feature-detect the class and probe the codec with isConfigSupported, and keep a fallback.