Re: [OT] collating sequences: using glibc

On Friday 9 May 2008 08:51, Steffen Schuler wrote:

> Perhaps the result of the GLIBC functions iswalpha and wprintf together
> with GNU sort explains better the collating of the alphabetic characters
> for the 3 locales C, en_US, en_US.utf8 (my current system: Debian
> GNU/Linux testing (lenny)):
> 
> $ cat collate.c
> #include <stdio.h>
> #include <wchar.h>
> #include <stdlib.h>
> 
> int
> main(int argc, char **argv)
> {
> wint_t i;
> 
> if (argc != 2) {
> fprintf(stderr, "usage: ./collate MAX_CHAR_CODE\n");
> exit(1);
> }
> 
> for (i = 1; i <= atoi(argv[1]); ++i)
> if (iswalpha(i))
> wprintf(L"%c\n", i);
> return 0;
> }
> $ cc -o collate collate.c
> $ export LC_ALL=C; ./collate 255 | sort | tr -d '\n'; echo
> ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
> $ export LC_ALL=en_US; ./collate 255 | sort | tr -d '\n'; echo
> aAbBcCdDeEfFgGhHiIjJkKlLmMnNoOpPqQrRsStTuUvVwWxXyYzZ
> $ # ^^^^^^^...: only Latin letters
> $ export LC_ALL=en_US.UTF-8; ./collate 65535 | sort | tr -d '\n'; echo
> aAbBcCdDeEfFgGhHiIjJkKlLmMnNoOpPqQrRsStTuUvVwWxXyYzZ
> $
> 
> 
> There seem to be some inconsistencies between gawk and glibc locale
> usage. Especially in the C-code sample only Latin letters are listed for
> LC_ALL=en_US contrary to the similar gawk sample.

Yes, I noticed that, but I could not explain it.

> Besides, the collating order may be different from the order of the
> character codes --- see:
> 
>     http://tinyurl.com/6gdraw

This is something that can immediately be noticed by looking at the en_US
output above.

> I'm really interested whether there are any further locale bugs in gawk
> (for trying to fix them).
> Perhaps I have more time for the research at the current weekend.

I'm not sure I can follow you on this, mostly due to my lack of in-depth
libc/wchar knowledge. However, I'll keep this program and come back to it
when my understanding of the subject is better.

Thanks!

-- 
All the commands are tested with bash and GNU tools, so they may use
nonstandard features. I try to mention when something is nonstandard (if
I'm aware of that), but I may miss something. Corrections are welcome.