unicodeunicode-normalization

In Unicode why क़ is excluded from composition, whereas Å is not?

I was going through Unicode Normalization Forms, and saw that devanagari letter क़ is excluded from composition under Script-specific Exclusions, whereas latin letter Å is not. What is the reason behind this?

Solution

  • The so-called script-specific composition exclusions are characters for which the generally preferred representation in actual usage is their fully decomposed form. The decisions about composition exclusions were all made in the late 90s so concrete information is hard to come by, but Robin Leroy on the Unicode mailing list managed to find two documents from 2001 that might explain why the Devanagari letters with nukta in the range U+0958..U+095F are excluded.

    L2/01-304 is a collection of feedback from various Indian experts. The section on Devanagari states in reference to the characters encoded at U+0958..U+095F:

    Usage of these code-points be discouraged, and rather use the corresponding DEVANAGARI LETTER followed by Nukta sign 093C to generate sounds in Urdu, Sindhi etc. For example

    0958 क़ ≡ 0915 क 093c ◌़

    L2/01-305 is a response to this feedback by Unicode representative Rick McGowan. It states in reference to the nukta letters:

    The document proposes to discourage the use of these precomposed characters with nuktas. By putting them into the composition exclusions list, UTC has already excluded them from the Form C normalization.

    It can be assumed that the Unicode Consortium was already aware of similar feedback from Indian delegates back when the composition exclusion list was being finalised. In the meeting notes from UTC #78 held in late 1998, Ken Whistler explains that Hebrew experts also prefer full decomposition, and Hebrew is another script whose letters with diacritics are composition-excluded, giving credence to this theory.

    The characters encoded at U+0958..U+095F did not exist as atomic letters in ISCII and weren’t strictly needed for roundtrip compatibility. As such there would have been little benefit for users of the Unicode Devanagari encoding to have the composed forms of these letters in their data.