Java Unicode considerations

Code points above U+FFFF are stored as a surrogate pair: a high surrogate (U+D800 to U+DBFF) followed by a low surrogate (U+DC00 to U+DFFF). Neither half is a valid character alone. Character.isHighSurrogate, isLowSurrogate and Character.toCodePoint(high, low) work with them.

Step through a string with an emoji and see both units form one pair.

1String s = "��";
2char hi = s.charAt(0);                            // '\uD83D'
3char lo = s.charAt(1);                            // '\uDE00'
4boolean pair = Character.isHighSurrogate(hi) && Character.isLowSurrogate(lo);   // true
5int cp = Character.toCodePoint(hi, lo);           // 0x1F600

1int units = s.length();                           // UTF-16 code units
2int count = s.codePointCount(0, units);           // real characters
3for (int i = 0; i < units; ) {
4    int cp = s.codePointAt(i);                    // combines a surrogate pair into one code point
5    i += Character.charCount(cp);                 // 2 for supplementary characters, otherwise 1
6}

Iterate code points rather than chars when the text may contain supplementary characters: use s.codePoints(), or a loop with codePointAt and Character.charCount. A plain for over charAt visits each half of a surrogate pair separately.

The animation advances by one or two units per step.

1String s = "a��b";
2int i = 0, chars = 0;
3while (i < s.length()) {
4    int cp = s.codePointAt(i);
5    chars++;
6    i += Character.charCount(cp);
7}                                                  // chars == 3, while s.length() == 4
8long letters = s.codePoints().filter(Character::isLetter).count();   // 2

1int units = s.length();                           // UTF-16 code units
2int count = s.codePointCount(0, units);           // real characters
3for (int i = 0; i < units; ) {
4    int cp = s.codePointAt(i);                    // combines a surrogate pair into one code point
5    i += Character.charCount(cp);                 // 2 for supplementary characters, otherwise 1
6}

Some characters can be written in more than one way: “é” is either one code point (U+00E9) or an “e” followed by a combining accent (U+0065 U+0301). The two look the same but are not equals. java.text.Normalizer converts text to a canonical form (NFC composes, NFD decomposes) so that comparisons and hashing agree. Normalize user input before comparing or storing it.

NFD followed by removing combining marks is a common way to strip accents.

1String a = "\u00E9";             // é as one code point
2String b = "e\u0301";            // e + combining acute accent
3boolean raw = a.equals(b);        // false
4boolean nfc = Normalizer.normalize(a, Normalizer.Form.NFC)
5        .equals(Normalizer.normalize(b, Normalizer.Form.NFC));   // true

toLowerCase() and toUpperCase() without an argument use the default locale, which changes the result in some languages: in Turkish, uppercase I lowercases to dotless ı, so "TITLE".toLowerCase() is not "title". A single character can also change length (German ß uppercases to SS). Pass Locale.ROOT (or Locale.ENGLISH) for protocol text, identifiers and keys.

Use the user's locale only for text shown to people.

1String t = "TITLE".toLowerCase(Locale.ROOT);                       // "title" everywhere
2String turkish = "TITLE".toLowerCase(new Locale("tr"));           // "tıtle" (dotless i)
3String up = "straße".toUpperCase(Locale.ROOT);                    // "STRASSE": length changed

Common wrong assumptions: one char is one character (emoji take two units); length() is the number of characters; reversing char by char keeps text valid (use StringBuilder.reverse, which keeps pairs intact); toUpperCase keeps the length; two equal-looking strings are equals (they may differ in normalization); one character always has one case form. For ASCII-only data, such as DNA or identifiers, the simple char loops are fine.

Decide up front whether your input is ASCII, and validate it if you rely on it.

1int units = s.length();                           // UTF-16 code units
2int count = s.codePointCount(0, units);           // real characters
3for (int i = 0; i < units; ) {
4    int cp = s.codePointAt(i);                    // combines a surrogate pair into one code point
5    i += Character.charCount(cp);                 // 2 for supplementary characters, otherwise 1
6}