Since it's time to post RfDs, I want to throw in the updated proposal for
the XCHAR wordset. I hope I have included all comments so far, and I also
included a reference implementation.

Problem:

ASCII is only appropriate for the English language. Most western
languages however fit somewhat into the Forth frame, since a byte is
sufficient to encode the few special characters in each (though not
always the same encoding can be used; latin-1 is most widely used,
though). For other languages, different char-sets have to be used,
several of them variable-width. Most prominent representant is
UTF-8. Let's call these extended characters XCHARs. Since ANS Forth
specifies ASCII encoding, only ASCII-compatible encodings may be
used. Furtunately, being ASCII compatible has so many benefits that
most encodings actually are ASCII compatible.

Proposal

Datatypes:

How hard would it be to extend the reference implemenation to UTF-32?

Erratum:

XC!+? ( xc xc_addr1 u1 -- xc_addr2 u2 flag ) XCHAR EXT
Stores the XCHAR xc into the buffer starting at address xc_addr1, u1
characters large. xc_addr2 points to the first memory location after
xc, u2 is the remaining size of the buffer. If the XCHAR xc did fit
into the buffer, flag is true, otherwise flag is false, and xc_addr2
u2 equal xc_addr1 u1. XC!+? is -save- +safe+ for buffer overflows, and
therefore preferred over XC!+.

Bernd Paysan writes:

^^^^^^^^
representation?

...

These appear to be subsections that should be put in informative
sections, not the normative "Proposal" section.

Bruce McFarling wrote:

UTF-32 is not ASCII compatible, unless you have a system where 1 CHAR = 32
bit.

Thanks, there was another save/safe error, as well.

Anton Ertl wrote:

Yes.

Moved it to an appendix

Bernd Paysan wrote:

[snipped]

Unfortunately, on first analysis, this is one proposal that Win32Forth
will not be adopting any time soon.

Windows is UTF-16, which is not ASCII compliant. Although Windows
provides APIs to translate from locale to locale, there is no method in
Win32Forth to automatically identify which parameters would be require
to be translated from XHCARS to UTF-16 and back; the programmer would be
responsible for coding the conversions.

We would need something like the proposal Anton made at EuroForth 2006
(http://dec.bournemouth.ac.uk/forth/euro/ef06/ertl06.pdf, A Portable C
Function Call Interface), with extensions to identify string pointers,
before implementing this.

Bernd Paysan writes:

Alex McDonald rivadpm.com> writes:

I don't see that you are any worse off with xchars in this situation
than with chars.

On Jul 16, 12:28 pm, an...@mips.complang.tuwien.ac.at (Anton Ertl)
wrote:

Anton Ertl wrote:

It's somewhat worse, because Windows has "A" prototypes, which convert the
current code page (can be multibyte) into UTF-16 on the fly. The "W"
prototypes take UTF-16 directly. But there's some light: UTF-8 is one of
the code pages in Windows (number 65001), and you can at least use
MultiByteToWideChar to convert data.

Actually, it might be possible to change the current code page to UTF-8, but
I didn't see a hint how to do that other than for console i/o (SetConsoleCP
and SetConsoleOutputCP). I must honestly admit that I don't like the online
access to MSDN represents information. The internal search is horrible, and
it's one of the rare sites where even Google is confused.