The product requirement sounds innocent:

No two users can have the same username.

That sentence hides a trap. It assumes a username is a string, and that two strings are the same when their bytes, code units, or code points are the same.

For internal database keys, maybe. For human-facing identifiers, not enough.

Unicode is doing something much larger than ASCII ever had to do. It represents many writing systems, combining marks, compatibility characters, invisible format controls, right-to-left text, and scripts whose letters can look like letters from other scripts. That is not a bug. It is the point of Unicode. But it means an identifier system needs a policy, not just an equality operator.

The username paypal and the username раураl are different strings. The second one starts with Cyrillic letters that look like Latin letters in many fonts. A database uniqueness constraint on raw strings will happily admit both. A human reader may not.

The engineering question is not:

are these code points equal?

It is:

what distinctions does this product promise users they can rely on?

Four Different Problems

Unicode security problems often get flattened into one phrase: homoglyph attack. That is too blunt. At least four different mechanisms matter.

Canonical equivalence. The same abstract character may have multiple Unicode representations. The character é can be a single precomposed code point or an e followed by a combining acute accent. UAX #15 is the Unicode annex that defines normalization forms; its summary says normalization gives equivalent strings a unique binary representation when implementations keep text in a normalized form.1 NFC is usually the first layer for stored display text.

Compatibility equivalence. Some characters are compatibility variants of others. The Kelvin sign K normalizes under NFKC to K; Roman numeral Ⅳ normalizes to IV. UAX #15 is careful here: compatibility equivalence can erase visual or semantic distinctions that matter in some contexts.1 That is why “normalize everything with NFKC” is not a universal answer. It is a policy choice.

Invisible and directional controls. A zero-width joiner inside an identifier can make the code-point sequence different without making the displayed string obviously different. Bidirectional controls can be worse in source code: Boucher and Anderson’s Trojan Source work showed how maliciously encoded source can appear different to a reviewer than to a compiler or interpreter.2 Unicode’s newer source-code guidance, UTS #55, treats lookalike glyphs, bidirectional reordering, and invisible characters as separate tooling and display problems.3 That attack is about source code, not usernames, but the lesson travels: visual review is not a parser.

Confusable scripts. Greek, Cyrillic, Latin, and other scripts contain characters that can be visually confusable. Unicode Technical Standard #39 defines security mechanisms for identifiers, including identifier profiles, confusable detection, mixed-script detection, and restriction levels.4 This is the layer that catches раураl against paypal.

Each mechanism has a different false-positive surface. That is why mature systems layer them instead of pretending one transform solves text.

A Small Registry Lab

The lab below has eight existing usernames and eight proposed candidates. Move the Case slider to choose the pair. Move Policy to see which comparison pipeline would block it.

The policy ladder is:

  1. raw equality;
  2. NFC;
  3. NFKC plus a small case-folding table;
  4. the same key after removing selected default-ignorable controls;
  5. a tiny UTS #39-style confusable skeleton plus a mixed-script guard.

This is intentionally not a production implementation of UTS #39. The real standard has data files, profiles, context rules, restriction levels, and versioning concerns. UAX #31 adds the other half of the story: identifier syntax is normally a profile, not a universal alphabet, and default-ignorable code points deserve explicit treatment.5 The browser lab uses a small readable table so the failure modes are visible.

Normalization Identifier key Blocked or reviewed Mixed script

Deterministic toy registry with a 573-assertion audit over nine cases: canonical equivalence, compatibility equivalence, case folding, invisible controls, bidirectional controls, mixed-script detection, and a small confusable skeleton. Production systems should use Unicode's full data files and a documented versioned profile.

The default case is éclair versus e plus a combining acute accent. Raw equality misses it. NFC catches it.

Move to compatibility letter. Raw equality and NFC miss Kilo; NFKC plus case folding collides it with kilo.

Move to invisible joiner. Even NFKC plus folding misses ad + U+200D + min; the identifier key catches it only after selected default-ignorable controls are removed.

Move to directional control. The candidate inserts U+202E, right-to-left override, inside admin. A registry that treats hidden format controls as part of the identity boundary lets the display layer and the comparison layer drift. The toy identifier key strips it; a real profile has to decide which such controls are allowed, rejected, or diagnosed.

Move to Cyrillic lookalike. Normalization does not save you. The candidate is already a perfectly normal Unicode string. It needs a script/confusable policy.

That progression is the point. A username policy is a pipeline:

decode -> normalize -> validate profile -> fold -> remove/restrict controls
       -> script policy -> confusable policy -> store stable comparison key

The exact order and strictness depend on the product. A programming-language compiler, a domain-name registry, a social network, and a private enterprise directory should not all use the same policy.

The False Positive Budget Is Product Design

It is tempting to make every identifier ASCII-only and declare victory. For some systems, that is a good answer. Internal service names, package names, and high-risk admin identifiers often benefit from a narrow alphabet.

For public names, ASCII-only can be exclusionary and commercially wrong. People and organizations have real names outside ASCII. The hard product work is deciding which distinctions users should be allowed to claim and which distinctions are too dangerous to sell as identity.

UTS #39 frames this as profiles, restriction levels, mixed-script detection, and confusable detection rather than a single ban list.4 IDNA2008 makes a similar point in a domain-specific way: internationalized domain names have their own protocol, definitions, and registry context, not just arbitrary Unicode labels pushed into DNS.6

For usernames, I would usually separate three stored values:

display_name      what the user chose and what other users see
comparison_key    normalized/folded/profiled key used for uniqueness
skeleton_key      spoofing key used for review, reservation, or blocking

The display name can preserve identity. The comparison key enforces ordinary uniqueness. The skeleton key protects other users from names that are distinct to the database but not reliably distinct to the eye.

The skeleton key should not silently rewrite the display name. It should decide whether registration is allowed, reserved, reviewed, or shown with warnings.

Versioning Is Part of the Schema

Unicode data changes over time. Standards evolve. UTR #36, for example, has been stabilized, and the Unicode page now points readers toward newer material such as UTS #39, UTS #55, and UAX #31 for some topics.7 That is a reminder: identifier policy is not a one-time regex.

If you operate a registry, store the policy version beside the derived keys:

username_display
username_key_v3
username_skeleton_v3
unicode_version
policy_version
review_status

Then migrations can be explicit. A new Unicode version may classify a character or confusable differently. A new business policy may reserve names that were previously allowed. You need to know whether an existing account is grandfathered, renamed, reviewed, or merely protected from future collisions.

The worst migration is the invisible one:

we changed normalization libraries and the uniqueness boundary moved

That is how a string bug becomes an identity bug.

What I Would Ship

For a serious identifier surface, I would want:

  1. a documented allowed-character profile;
  2. one canonical comparison key for uniqueness;
  3. a confusable skeleton key for high-risk collision review;
  4. explicit handling of default-ignorable and bidirectional controls;
  5. script-mixing rules that differ by product and locale;
  6. UI that reveals code points or warnings when a name is risky;
  7. migrations that record Unicode and policy versions.

The key design smell is using raw equality as the boundary while showing users a rendered string as the identity.

Rendered text is an interface. Unicode code points are a representation. Identity is a policy.

Treat the username like a security object, not a string.

Works Cited

  1. Unicode Consortium, “Unicode Standard Annex #15: Unicode Normalization Forms.” UAX #15. ↩ ↩2

  2. Nicholas Boucher and Ross Anderson, “Trojan Source: Invisible Vulnerabilities,” USENIX Security 2023; first disclosed in 2021. USENIX PDF, arXiv. ↩

  3. Unicode Consortium, “Unicode Technical Standard #55: Unicode Source Code Handling.” UTS #55. ↩

  4. Unicode Consortium, “Unicode Technical Standard #39: Unicode Security Mechanisms.” UTS #39. ↩ ↩2

  5. Unicode Consortium, “Unicode Standard Annex #31: Unicode Identifiers and Syntax.” UAX #31. ↩

  6. John Klensin, “Internationalized Domain Names for Applications (IDNA): Definitions and Document Framework,” RFC 5890, IETF, 2010. RFC 5890. See also RFC 5891 for the IDNA protocol. RFC 5891. ↩

  7. Unicode Consortium, “Unicode Security Considerations” stabilized report page, noting that UTR #36 is no longer maintained and that some material has been superseded by newer standards. UTR #36. ↩