Re: Supporting full Unicode

[David Starner <dvdeug@email.ro>]

On Wed, 12 May 2004 14:53:23 +0000, Bj�rn Persson wrote:

> Looks troublesome, eh? For UTF-8 I don't think it's even possible to 
> define such a type. I'd rather just define UTF-16 and UTF-8 strings as 
> byte sequences and represent even single characters as strings.

Why would encode UTF-16 as a byte sequences when they could encode it as
a series of words? You can't use UTF-16 internally as byte sequences
without worry about byte-order marks, because UTF-16 is constructively
ambigious as to whether it's big-endian or little-endian. Anything you
defined would either be UTF-16BE or UTF-16LE, and spend a lot of time
reassembling character pieces on the wrong endian architecture. UTF-16
should usually be encoded as words.

As for characters, they're not much use with Unicode. Even with Latin-1,
you can't uppercase a character to character, and any system that does
it is wrong, include Ada. The German esszett (�) uppercases to SS. You
can't even hold a whole "character" in a character in Unicode, because
you can't fit any attached combining characters in.