Re: Supporting full Unicode

[David Starner <dvdeug@email.ro>]

> Indeed UTF-8 seems to rule. Probably because there are more ready-to-use low
> level tools for 8-bit characters. Actually the proper tools for Unicode
> should be 24-bit based. An ugly fact about Unicode is that the code space is
> 24-bit and the encodings are all but 24 (8, 16, 32).

Why is that ugly? UTF-16 or UTF-8 is virtually always going to be smaller,
unless most of your text is in an obscure dead tongue, which is unlikely
to found in quantities that need compression. It's not going to be faster
to process, unless you're running on some terribly obscure architecture
that natively handles 24 bit words.

As someone else pointed out, it's not 24, it's roughly 20.1. 

As for compression, a comparison of compression formats on various Unicode
encodings was made[1], and it was found that most of the difference
between encodings was wiped out by compression.

[1] http://www.cs.fit.edu/~ryan/compress/