Unicode, String internals

Every JavaScript string is Unicode text. Unicode assigns each character a number called a code point, and those code points span a huge range — far more than the letters on your keyboard. The trouble is that JavaScript stores strings in a fixed-width unit that predates the full Unicode range, and that mismatch is the source of nearly every surprise in this article.

Let’s start with the easy part: putting a specific character into a string by its code.

Escaping characters by code

You don’t have to type a character directly. You can write its hexadecimal Unicode code using one of three escape notations. Each covers a different slice of the range.

\xXX — two hex digits, a value from 00 to FF. That covers only the first 256 code points, which is where the Latin alphabet, most punctuation, and a handful of common symbols live. Beyond FF this notation can’t reach.

alert( "\x48" ); // H   (U+0048)
alert( "\xB5" ); // µ   the micro sign

\uXXXX — exactly four hex digits, 0000 to FFFF. This reaches the whole common range, which is what you’ll use most of the time. Codes above FFFF can still be written this way, but only by spelling out a surrogate pair (two \uXXXX escapes back to back), which we’ll get to shortly.

alert( "µ" ); // µ   same character as \xB5, just written with 4 digits
alert( "ω" ); // ω   a Greek letter
alert( "→" ); // →   the rightward arrow

\u{X…XXXXXX} — the braces let you drop in anywhere from one to six hex digits, up to 10FFFF, which is the highest code point Unicode defines. This is the clean way to write anything, including characters far beyond FFFF, without thinking about surrogate pairs.

alert( "\u{20BB7}" ); // 𠮷   a rare Chinese character (a high code point)
alert( "\u{1F680}" ); // 🚀   a rocket (also high)

Why surrogate pairs exist

Here’s the historical knot. When JavaScript was designed, Unicode fit in 16 bits, so the language stored each character as a single 2-byte unit using UTF-16. Two bytes give 65,536 (2**16) possible values — enough at the time.

The common characters still fit in those two bytes: Latin and Cyrillic and Greek letters, digits, punctuation, and the core CJK ideographs (CJK being Chinese, Japanese, and Korean script). Each of those is one 4-hex-digit code, one storage unit.

But Unicode outgrew 16 bits. Emoji, rarer ideographs, and specialized mathematical letters all sit above FFFF, and there’s no room for them in a single 2-byte unit. UTF-16 handles this by encoding one such character as two 2-byte units — a surrogate pair.

The side effect you’ll notice first: length counts storage units, not characters. So a single high character reports a length of 2.

alert( '𝄞'.length ); // 2   MUSICAL SYMBOL G CLEF
alert( '🚀'.length ); // 2   ROCKET
alert( '𠮷'.length ); // 2   a rare Chinese character

Each of those strings holds one symbol as far as a human is concerned, yet length says 2. Surrogate pairs didn’t exist when the language was born, so a lot of built-in behavior treats each half as if it were its own character.

Indexing shows the same crack. Reach into a surrogate pair by position and you pull out half of it — a code unit that means nothing on its own.

alert( '𝄞'[0] ); // a lone, meaningless half of the pair
alert( '𝄞'[1] ); // the other lone half

Neither half is a real character; displayed alone they render as garbage or a replacement box.

Detecting and reading surrogate pairs

The two halves aren’t random. The standard reserves a block of codes exclusively for them:

• the first (high) half always falls in 0xD800..0xDBFF
• the second (low) half always falls in 0xDC00..0xDFFF

Those ranges are reserved for nothing else, so you can recognize a surrogate half purely from its code.

To work with whole code points instead of raw units, JavaScript added String.fromCodePoint and str.codePointAt. They mirror the older String.fromCharCode and str.charCodeAt, with one difference: the newer pair understands surrogate pairs and the older pair does not.

// charCodeAt reads a single unit, so at position 0 it returns only the first half of 𝄞:
alert( '𝄞'.charCodeAt(0).toString(16) ); // d834

// codePointAt sees both halves and returns the real code point:
alert( '𝄞'.codePointAt(0).toString(16) ); // 1d11e

The .toString(16) here just prints the code in hex so it lines up with the U+... notation.

Ask for position 1 — which lands in the middle of the pair, so it’s the wrong thing to do — and both methods can only hand back the lone second half:

alert( '𝄞'.charCodeAt(1).toString(16) ); // dd1e
alert( '𝄞'.codePointAt(1).toString(16) ); // dd1e
// the meaningless low half by itself

The Iterables chapter shows another angle: a for..of loop and the spread syntax step over strings by code point, not by unit, so they skip over this whole problem for reading. There are dedicated libraries for heavy Unicode work too, though none common enough to single out.

Diacritical marks and normalization

Many writing systems build letters from a base character plus a mark above or below it — accents, dots, tildes. The letter a alone spawns a whole family: àáâäãåā.

Unicode gives the most common of these composites their own single code point. But it can’t possibly pre-assign a code to every base-plus-marks combination, so it also supports building them on the fly: write the base character, then follow it with one or more combining mark characters that decorate it.

Put a next to the “combining circumflex” (̂) and it renders as a single â:

alert( 'â' ); // â

Need another mark? Stack it on. Add “combining dot below” (̣) and you get an a wearing a circumflex above and a dot below:

alert( 'ậ' ); // ậ

The flexibility comes with a trap. Two strings can look pixel-for-pixel identical yet be built from different sequences, because the marks were added in a different order:

let s1 = 'ậ'; // ậ  =  a + circumflex + dot below
let s2 = 'ậ'; // ậ  =  a + dot below + circumflex

alert( `s1: ${s1}, s2: ${s2}` ); // they render identically

alert( s1 == s2 ); // false (?!) — different order, so different data

Comparison works on the underlying code units, and those differ, so == says false even though your eyes say otherwise. This bites string comparisons, deduplication, and search.

The cure is Unicode normalization: an algorithm that rewrites any string into one canonical form, so that anything meant to be equal ends up byte-identical. It’s exposed as str.normalize().

alert( "ậ".normalize() == "ậ".normalize() ); // true

There’s a nice twist in this particular case. Normalizing collapses the three-character sequence into a single code point, ậ (a precomposed “a with circumflex and dot below”), because that composite is common enough that Unicode gave it a dedicated code:

alert( "ậ".normalize().length ); // 1
alert( "ậ".normalize() == "ậ" ); // true

Don’t expect that collapse every time — it happens only when a precomposed code point exists for the combination. Many base-plus-marks sequences normalize to a shorter but still multi-character form.

If you want the full rules and the differences between the forms, they’re specified in the Unicode standard’s Normalization Forms annex. For everyday work, knowing that normalization exists and reaching for .normalize() before comparing accented text will carry you a long way.