Re: Ada 202x; 2022; and 2012 and Unicode package (UTF-nn encodings handling)

["Lawrence D’Oliveiro" <ldo@nz.invalid>]

On Tue, 2 Sep 2025 18:20:09 -0600, Alex // nytpu wrote:

> (This whole mess is because UCS-2 and the UCS-2->UTF-16 transition was a
> hellish mess caused by extreme lack of foresight and it's horrible they
> saddled everyone, including people not using UTF-16, with this crap.

I gather the basic problem was that Unicode was originally going to be a 
fixed-length 16-bit code, and that was that. And so early adopters 
(Windows NT and Java among them), built UCS-2 right into their DNA.

Until Unicode 2.0, I believe it was, where they went “on second thought, 
let’s go beyond our original brief and start including all kinds of other 
things as well” ... and UCS-2 had to become UTF-16 ...

> UTF-16 and its surrogate pairs is also what's responsible for the
> maximum scalar value being 0x0010_FFFF instead of 0xFFFF_FFFF even
> though UTF-32 and UTF-8 and even goddamn UTF-EBCDIC and the weird
> encoding the Chinese government came up with can all trivially encode
> full 32-bit values)

I wondered about that limit ...

> Plus conveniently Ada doesn't have routines for normalization, but can't
> hold that against it since neither does any other programming language
> because the lookup tables required are like 20 MiB even when optimized
> for space.

I think Python has them
<https://docs.python.org/3/library/unicodedata.html>. But then, on 
platforms with decent package management, that data can be shared with 
other installed packages that require it as well.

> Plus you shouldn't normalize text other than performing actions like
> substring matching, equality tests, or sorting---and even if you
> normalize when performing those, *when possible* you should store the
> unnormalized original for display/output afterwards.

I thought it was always safe to store decomposed versions of everything.