Re: Supporting full Unicode

["Björn Persson" <spam-away@nowhere.nil>]

Ludovic Brenta wrote:

> Bjorn Persson wrote:
> 
>>David Starner wrote:
>>
>>>they should have defined Wide_Character to be UTF-16 like Java did.
>>
>>Keeping in mind that in UTF-16 some characters take two bytes and
>>others take four, how do you propose to define that type?

[...]

> But UTF-8 is gaining momemtum.  Originally intended as an external
> encoding only, it is now in use as an internal encoding, too.

[...]

I'm not trying to stop anyone from using whatever encoding they like 
internally. Just make sure you always know which encoding you have.

I just asked how David wanted to define a UTF-16 *character* type. A 
UTF-16 *string* can be represented as an array of 16-bit elements, but 
those elements aren't characters. In some cases an element is just half 
a character. Only the fixed-width encodings can be easily represented as 
arrays of characters.

Here's a datatype that can represent all UTF-16 characters and doesn't 
accept illegal characters:

type Subrange is (One_Byte_Below_Hole,
                   One_Byte_Above_Hole,
                   Two_Byte);
type Code_Point_Below_Hole is range 0 .. 16#D7FF#;
type High_Surrogate_Code_Point is range 16#D800# .. 16#DBFF#;
type Low_Surrogate_Code_Point is range 16#DC00# .. 16#DFFF#;
type Code_Point_Above_Hole is range 16#E000# .. 16#FFFD#;
type UTF_16_Character(Block : Subrange) is record
    case Block is
       when One_Byte_Below_Hole =>
          Value_Below_Hole : Code_Point_Below_Hole;
       when One_Byte_Above_Hole =>
          Value_Above_Hole : Code_Point_Above_Hole;
       when Two_Byte =>
          High_Surrogate_Value : High_Surrogate_Code_Point;
          Low_Surrogate_Value : Low_Surrogate_Code_Point;
    end case;
end record;

Looks troublesome, eh? For UTF-8 I don't think it's even possible to 
define such a type. I'd rather just define UTF-16 and UTF-8 strings as 
byte sequences and represent even single characters as strings.

So go ahead and use UTF-8 in your programs, or Shift-JIS or EBCDIC for 
all I care, but think twice before you define datatypes for 
variable-width *characters*.

-- 
Björn Persson

jor ers @sv ge.
b n_p son eri nu