Why Emoji Skin Tone Modifiers Break Text Comparison

Two reactions on a message are a thumbs-up with different skin tones. They compare unequal, which is expected. They also each report as containing the plain yellow thumbs-up, which is not, and a reaction counter written with includes() merges all six into one bucket while an equality-based one splits them into six.

What the sequence actually is

An emoji with a skin tone is not a character. It is a sequence: a base emoji code point followed by one of five modifier code points, U+1F3FB through U+1F3FF, named EMOJI MODIFIER FITZPATRICK TYPE-1-2 through TYPE-6. The renderer sees the pair and draws one glyph. Nothing about the encoding merges them.

Plain thumbs up  "👍"
  code points  U+1F44D
  UTF-8        F0 9F 91 8D            (4 bytes)
  UTF-16       D83D DC4D              (2 code units — a surrogate pair)
  JS .length   2

Medium tone      "👍🏽"
  code points  U+1F44D  U+1F3FD
  UTF-8        F0 9F 91 8D  F0 9F 8F BD   (8 bytes)
  UTF-16       D83D DC4D  D83C DFFD       (4 code units)
  JS .length   4

The five modifiers
  U+1F3FB  light          U+1F3FC  medium-light
  U+1F3FD  medium         U+1F3FE  medium-dark
  U+1F3FF  dark

The .length values are the first surprise for anyone who has not looked before. Even the plain emoji is 2 in JavaScript, Java and C#, because it is outside the Basic Multilingual Plane and needs a surrogate pair in UTF-16. Indexing with [0] gives you a lone high surrogate, which is not a valid character and will render as a replacement glyph or throw when re-encoded.

Why one contains the other

The modifier is appended. That single structural fact produces the whole bug class.

const plain  = "\u{1F44D}";              // 👍
const medium = "\u{1F44D}\u{1F3FD}";     // 👍🏽
const dark   = "\u{1F44D}\u{1F3FF}";     // 👍🏿

plain === medium          // false — expected
medium.includes(plain)    // true  — surprising
medium.startsWith(plain)  // true  — surprising
medium.includes(dark)     // false

// So a reaction tally keyed by "does the string contain 👍"
// counts all six variants as one, while a tally keyed by
// equality reports six unrelated reactions.

[...medium].length              // 2 — code points, still not 1
[...new Intl.Segmenter("en", { granularity: "grapheme" })
   .segment(medium)].length     // 1 — the number a person would give

The asymmetry is what makes it hard to spot in review. A test that checks “plain does not equal toned” passes. A test that checks “toned does not contain a different tone” passes. A test that checks “toned does not contain plain” is the one nobody writes, and it is the only one that fails.

Real consequences, in rough order of how often they appear: a moderation filter that matches a banned emoji by substring fires on every toned variant, which may be intended or may not; a validation rule of “exactly one emoji” implemented as a length check rejects every toned emoji; a fixed-width truncation cuts between base and modifier, leaving a bare modifier that renders as a coloured square; and a database column sized in characters overflows on input that looked like one character.

Normalisation does not fix this

The instinct after reading about combining marks is to reach for normalize(). It does nothing here, and understanding why is what stops you reaching for it again.

Canonical decomposition operates on characters that have a canonical decomposition mapping in the Unicode Character Database. Emoji modifiers have none. They are not combining marks in the general-category sense — U+1F3FD is category Sk (Symbol, modifier), not Mn — so /\p{Mn}/u does not match them and no normalisation form touches them. NFC, NFD, NFKC and NFKD all leave a toned emoji exactly as it arrived.

const medium = "\u{1F44D}\u{1F3FD}";
medium.normalize("NFKD") === medium   // true — nothing changed

// Same for variation selectors:
const heart      = "\u2764";          // ❤  text presentation
const heartEmoji = "\u2764\uFE0F";    // ❤️ emoji presentation
heart === heartEmoji                             // false
heart.normalize("NFC") === heartEmoji.normalize("NFC")  // still false

The variation-selector case is the one that costs most debugging time, because U+FE0F VARIATION SELECTOR-16 is completely invisible in every editor. Two hearts pasted from two sources, one with the selector and one without, are different strings that render identically on most platforms and differently on some. Nothing normalises them together.

ZWJ sequences and variation selectors

Skin tone is the simplest case. The same structural problem scales badly through joined sequences.

• ZWJ sequences. A family emoji is several people joined by U+200D ZERO WIDTH JOINER: woman, ZWJ, woman, ZWJ, girl, ZWJ, boy. Eleven code points for one glyph, and it contains the single woman emoji as a substring twice.
• Combinations. A health worker with a skin tone is base, modifier, ZWJ, the medical symbol, and then U+FE0F to force emoji presentation on the symbol. Five code points, four of which are prefixes of something else.
• Flags. A country flag is two regional indicator symbols, so a string of two flags contains, as a substring, a third flag made from the second half of the first and the first half of the second.
• Keycaps. A keycap digit is the ASCII digit, U+FE0F, and U+20E3. So 1️⃣ genuinely starts with the character 1, and a numeric parse of the string succeeds.

Unicode publishes the full sequence inventory as data files rather than leaving it to be inferred; the Unicode Technical Standard #51, Unicode Emoji defines the modifier and ZWJ sequence grammar and lists the valid combinations.

The fix

1. Stop comparing emoji with includes() or startsWith(). Substring matching on emoji is almost never what was meant, and the cases where it looks right are coincidences of encoding.
2. Segment before you do anything per-character. Use Intl.Segmenter with grapheme granularity in JavaScript, BreakIterator in Java or ICU, or a grapheme library in Python and Go. Counting, truncating and reversing are all grapheme-level operations.
3. Decide, as a product question, whether skin tone is part of the identity of a reaction. If it is, key on the full sequence. If it is not, define an explicit fold.
4. Write that fold as a deliberate strip of U+1F3FB–U+1F3FF and U+FE0F, applied only to the key. Keep the original sequence for display, so a person still sees the tone they picked.
5. Store both. A reaction_key column for grouping and a reaction_raw column for rendering costs one column and removes the entire ambiguity.
6. Check the database can hold emoji at all. MySQL’s utf8 is a three-byte encoding and cannot store any character outside the BMP; the symptom is Incorrect string value: '\xF0\x9F\x91\x8D' for column. The column and the connection charset both need to be utf8mb4.
7. Truncate on grapheme boundaries. Cutting a sequence in the middle produces a lone modifier or a dangling ZWJ, and some clients render that as a visible tofu box next to text that otherwise looks fine.