Parrot assumes command line arguments are ASCII

[pmichaud]

Currently Parrot (incorrectly) assumes that all command line arguments are ASCII:

$ cat x.pir
.sub 'main'
    .param pmc args

    $S0 = args[1]
    say $S0

    $I1 = charset $S0
    $S1 = charsetname $I1
    say $S1

    $I1 = encoding $S0
    $S1 = encodingname $I1
    say $S1

    $I1 = length $S0
    say $I1
.end
$ ./parrot x.pir 'say «hello»'
say «hello»
ascii
fixed_8
13
$ 

Most would expect the above to be a unicode/utf8 string of length 11.

If Parrot itself cannot be easily changed to accept unicode/utf8 command line arguments, then it would be nice to have a way to easily convert the "ascii" strings in args into proper unicode strings. Thus far I've been unable to find a good way of doing that.

This ticket also relates to Rakudo RT #66364.

Pm

[chromatic]

Encode command-line arguments as Unicode, not ASCII

[chromatic]

This patch passes all Parrot tests; how does it work for Rakudo?

[pmichaud]

Everything works great in Rakudo now. We should add whatever tests we want for this, and then I think we can mark it as "fixed".

Pm

[jkeenan]

pmichaud, chromatic:

Is this still a live issue? If so, the patch will have to be re-pulled, because the area where it presumably would apply in src/embed.c has changed significantly.

PARROT_CANNOT_RETURN_NULL
static PMC*
setup_argv(PARROT_INTERP, int argc, ARGIN(const char **argv))
{
...
    for (i = 0; i < argc; ++i) {
        /* Run through argv, adding everything to @ARGS. */
        STRING * const arg = Parrot_str_new_init(interp, argv[i], 
                strlen(argv[i]),
                Parrot_utf8_encoding_ptr, PObj_external_FLAG);

        if (Interp_debug_TEST(interp, PARROT_START_DEBUG_FLAG))
            Parrot_io_eprintf(interp, "\t%vd: %s\n", i, argv[i]);

        VTABLE_push_string(interp, userargv, arg);
    }
...
}

[whiteknight]

I suspect this is either fixed or on the way to being fixed with recent work by nwellnhof and plobsing and others. I would like to hear what they have to say about it.

[nwellnhof]

This was fixed by plobsing for Linux only. For other Unices, we need code to initialize the platform encoding. For Windows it's not that easy. See here for our discussion on IRC:  http://irclog.perlgeek.de/parrot/2011-01-09#i_3167023