t/library/pcre.t: Test #2 failing on MSWin32

[jkeenan]

This is a test which is consistently failing in Smolder reports being submitted from MSWin32 systems. (Granted, it looks as if only fperrad is smoking on Windows right now.)

1..2
ok 1 - libpcre loading
not ok 2 - soup to nuts

#   Failed test 'soup to nuts'
#   at t/library/pcre.t line 104.
#          got: 'ok 1
# ok 2
# not ok 3
# not ok 4
# not ok 5
# '
#     expected: 'ok 1
# ok 2
# ok 3
# ok 4
# ok 5
# '
# Looks like you failed 1 test of 2.

Until we definitively fix it, we can't be 100% confident that other patches/tickets awaiting application/resolution (e.g., TT #886) are working.

There is some very Win32-specific code in t/library/pcre.t, so it will really need the attention of someone who can work on that OS. Any takers?

Thank you very much.
kid51

[jkeenan]

Still failing as of r43419: see  this Smolder report.

[jkeenan]

As reported in TT #886, in the last three days Ron Blasch++ has submitted lots of Smolder reports on various Win32 platforms. While he is getting some test failures in most of these submissions, he is not getting failures in t/library/pcre.t.

[fperrad]

Ron Blasch has no failures in t/library/pcre.t, because these tests are skipped. See for example  http://smolder.plusthree.com/app/projects/tap_stream/31672/312.

pcre is not available (not installed ?) on its platform.

[jkeenan]

Replying to fperrad:

Thanks for catching that, François.

Do you know why the test was/is failing on your Win32 smoke tests?

Thank you very much.

kid51

[fperrad]

As far I remember, this test begun to fail at the end of September, after some change in NCI subsystem (tools/build/nativecall.pl ?).

See below, a detailed output of tests :

> parrot t\library\pcre_1.pir
Loaded

> parrot t\library\pcre_2.pir
ok 1
ok 2
not ok 3
not ok 4
not ok 5

[Util]

The above problem is now occurring on Darwin/Intel 10.5, Parrot r43484.

Also, when run via t/harness, a warning is issued by malloc.

$ perl t/harness --gc-debug t/library/pcre.t
t/library/pcre.t .. 1/2 
#   Failed test 'soup to nuts'
#   at t/library/pcre.t line 104.
#          got: 'ok 1
# ok 2
# parrot(23341) malloc: *** error for object 0xa38984: Non-aligned pointer being freed
# *** set a breakpoint in malloc_error_break to debug
# not ok 3
# not ok 4
# not ok 5
# '
#     expected: 'ok 1
# ok 2
# ok 3
# ok 4
# ok 5
# '
# Looks like you failed 1 test of 2.
t/library/pcre.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/2 subtests 

Test Summary Report
-------------------
t/library/pcre.t (Wstat: 256 Tests: 2 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
Files=1, Tests=2,  0 wallclock secs ( 0.03 usr  0.01 sys +  0.08 cusr  0.05 csys =  0.17 CPU)
Result: FAIL

$ ./parrot --gc-debug t/library/pcre_2.pir 
ok 1
ok 2
ok 3
not ok 4
not ok 5

[Util]

Backtrace as requested by chromatic:

$ gdb ./parrot
GNU gdb 6.3.50-20050815 (Apple version gdb-967) (Tue Jul 14 02:11:58 UTC 2009)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-apple-darwin"...Reading symbols for shared libraries ........... done

(gdb) run -D40 t/library/pcre_2.pir
Starting program: /Users/bruce/Perl/Parrot/Release_20_test/parrot/parrot -D40 t/library/pcre_2.pir
Reading symbols for shared libraries ++++++++++......... done
ok 1
Reading symbols for shared libraries . done
ok 2
parrot(23852) malloc: *** error for object 0xa38984: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
not ok 3
not ok 4
not ok 5

Program exited normally.
(gdb) break malloc_error_break
Breakpoint 1 at 0x94ef64a9
(gdb) run -D40 t/library/pcre_2.pir
Starting program: /Users/bruce/Perl/Parrot/Release_20_test/parrot/parrot -D40 t/library/pcre_2.pir
Reading symbols for shared libraries . done
ok 1
Reading symbols for shared libraries . done
ok 2
parrot(23854) malloc: *** error for object 0xa38984: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug

Breakpoint 1, 0x94ef64a9 in malloc_error_break ()
(gdb) bt
#0  0x94ef64a9 in malloc_error_break ()
#1  0x94ef1497 in szone_error ()
#2  0x94e1b523 in szone_free ()
#3  0x94e1b38d in free ()
#4  0x0049d31e in mem_sys_free (from=0xa38984) at src/gc/alloc_memory.c:325
#5  0x003fcaa8 in Parrot_str_free_cstring (p=0xa38984 "reference to non-existent subpattern") at src/string/api.c:2535
#6  0x004e7a3a in pcf_p_tiB3P (interp=0x901860, self=0xa0c848) at src/nci.c:12615
#7  0x005f2931 in Parrot_NCI_invoke (interp=0x901860, pmc=0xa0c848, next=0x271c0) at nci.pmc:338
#8  0x003ffda2 in Parrot_invokecc_p (cur_opcode=0x271b8, interp=0x901860) at core.ops:433
#9  0x0050a3ce in runops_slow_core (interp=0x901860, runcore=0x90b420, pc=0x271b8) at src/runcore/cores.c:848
#10 0x00508d1a in runops_int (interp=0x901860, offset=0) at src/runcore/main.c:546
#11 0x004b2ee8 in runops (interp=0x901860, offs=0) at src/call/ops.c:99
#12 0x004a900c in Parrot_pcc_invoke_from_sig_object (interp=0x901860, sub_obj=0xa0bc90, call_object=0xa0bcb8) at src/call/pcc.c:314
#13 0x004a9400 in Parrot_pcc_invoke_sub_from_c_args (interp=0x901860, sub_obj=0xa0bc90, sig=0x6d3584 "P->") at src/call/pcc.c:75
#14 0x0048ccd1 in Parrot_runcode (interp=0x901860, argc=1, argv=0xbffff69c) at src/embed.c:825
#15 0x006a137d in imcc_run_pbc (interp=0x901860, obj_file=0, output_file=0x0, argc=1, argv=0xbffff69c) at compilers/imcc/main.c:792
#16 0x006a2019 in imcc_run (interp=0x901860, sourcefile=0xbffff788 "t/library/pcre_2.pir", argc=1, argv=0xbffff69c) at compilers/imcc/main.c:1075
#17 0x00002399 in main (argc=1, argv=0xbffff69c) at src/main.c:60
(gdb) 

[ronaldws]

I seemed to get the whole test file to run in it's entirety on Win32 with pcre and good bit of work.

I had trouble getting parrot to recognize my installation of the pcre7.0 win32 binary so I put in a brand new installation of mingw and msys and used that to install pcre8.10 from source. I work with ActiveState Perl and mingw on Vista. "make install" for the pcre installation seemed to put the pcre files in c:\MinGW\msys\1.0\local\{bin,lib,...etc...} so I just copied those files into the corresponding c:\MinGW directories. Since prove -v t\library\pcre.t still didn't find the pcre library a few days of hacking around followed. Eventually I stumbled on the idea of copying a 'dll' file named c:\MinGW\bin\libpre-0.dll to c:\MinGW\bin\pcre3.dll and the test started to run. Better than that I removed all the todo references in test 2 and both tests still passed.

I am still working on the area but thought this experience might be of help to someone.

[ronaldws]

Have pcre.pm report back "no" if no pcre rather than done

[ronaldws]

remove win32 todo logic since tests, at least sometimes, run on win32

[ronaldws]

Add library name genned by install of pcre 8.10 from source.

[ronaldws]

It looks to me like test 2 can run on Win32 in some of my configurations including the parts that were marked todo. Since it can fail on Darwin and runs for me on Win32 I would recommend renaming the ticket with a subject like "t/library/pcre.t: Test #2 failing with some configurations". Old failures should hopefully be retested and any reported failures, whatever the OS, should include detailed version and environment information for both OS and build tools.

There were some hiccups with the pcre binary and a back version of MinGW including gcc 3.4 but actually I eventually got that to run too. When I upgraded to the current MinGW with gcc 4.5, I got both the pcre 7.0 binary install and the pcre 8.10 install from source working and running both tests without any "todo" logic for test 2.

There were some issues along the way. First I was confused, as I think some others on this thread seemed to be, when "perl Config.pl" reported back "done" instead of "no" when the parrot configuration was built without a pcre present. I have included a patch to config\auto\pcre.pm that flatly reports back "no" when there is no "pcre" present. While getting parrot to recognize pcre 8.10 could be hacked with a copy of a dll named pcre3.dll, the attached patch recognizes the pcre 8.10 library directly. The patch to the test file, pcre.t, removes the "todo" logic since the logic was tied to win32 but wasn't needed at all on my vista system. Along the way I learned that "make install prefix=/c/MinGW" for the pcre make put the pcre files where they needed to be on my system.

[jkeenan]

Replying to ronaldws:

I applied the pcre.pm.patch in r48971. The others look like they should be tested on a couple of other machines before applying.

Ron, thanks for the very detailed exploration of these issues.

kid51

[fperrad]

Currently, my platform is WinXP, with Strawberry Perl 5.12.1 which includes MinGW/gcc 4.4.3.

I've installed a PcRe 7.0 binaries from GnuWin32.net.

I obtain the following output :

> parrot t/library/pcre_1.pir
Loaded

> parrot t/library/pcre_2.pir
ok 1
ok 2

And gdb doesn't catch an exception :

> gdb -args parrot.exe t/library/pcre_2.pir
GNU gdb (GDB) 7.1
(gdb) r
ok 1
ok 2
ok 3
not ok 4
not ok 5

Program exited normally.
(gdb)

[jkeenan]

Replying to fperrad:

François,

Do you think we should apply the pcre.t.patch and the pcre.pir.patch?

Thanks.

kid51

[fperrad]

pcre.pir.patch applied in r48995.

[ronaldws]

Replying to fperrad:

Currently, my platform is WinXP, with Strawberry Perl 5.12.1 which includes MinGW/gcc 4.4.3.

I have a WinXP machine and I installed Strawberry Perl 5.12.1 with gcc 4.4.3.

I've installed a PcRe 7.0 binaries from GnuWin32.net.

Meaning gnuwin32.sourceforge.net ? I also installed the PCRE 7 binaries.

I obtain the following output : ...

gdb -args parrot.exe t/library/pcre_2.pir

GNU gdb (GDB) 7.1 (gdb) r ok 1 ok 2 ok 3 not ok 4 not ok 5

When I run ".\parrot t/library/pcre_2.pir" all 5 tests pass ok. Still wondering how to debug this.

[ronaldws]

Replying to ronaldws:

Replying to fperrad:

I obtain the following output : ...

gdb -args parrot.exe t/library/pcre_2.pir

GNU gdb (GDB) 7.1 (gdb) r ok 1 ok 2 ok 3 not ok 4 not ok 5

Still wondering how to debug this.

I now have an idea on how to debug the problem. The "not ok 4" indicates that the parrot pcre library match function called the pcre library pcre_exec function which returned a negative value. The negative value is likely to be an error code which we might be able to decipher. The pcre.t.dbg_patch file alters the the test to print out the return code when it fails.

Mr. Perrad, please apply the pcre.t.dbg_patch patch on your system, run "prove -v t\library\pcre.t" and post the output.

Thanks, Ron

[fperrad]

Replying to ronaldws:

Replying to ronaldws:

Replying to fperrad:

I obtain the following output : ...

gdb -args parrot.exe t/library/pcre_2.pir

GNU gdb (GDB) 7.1 (gdb) r ok 1 ok 2 ok 3 not ok 4 not ok 5

Still wondering how to debug this.

I now have an idea on how to debug the problem. The "not ok 4" indicates that the parrot pcre library match function called the pcre library pcre_exec function which returned a negative value. The negative value is likely to be an error code which we might be able to decipher. The pcre.t.dbg_patch file alters the the test to print out the return code when it fails. Mr. Perrad, please apply the pcre.t.dbg_patch patch on your system, run "prove -v t\library\pcre.t" and post the output.

I obtain :

> prove -v t\library\pcre.t
t\library\pcre.t ..
1..2
ok 1 - libpcre loading

not ok 2 - soup to nuts
#   Failed test 'soup to nuts'
#   at t\library\pcre.t line 104.
#          got: 'ok 1
# ok 2
# not ok 3
# ok value seems to be error code - value is: -2
# not ok 4
# not ok 5
# '
#     expected: 'ok 1
# ok 2
# ok 3
# ok 4
# ok 5
# '
# Looks like you failed 1 test of 2.

Thanks, Ron

[ronaldws]

Replying to fperrad:

Replying to ronaldws:

Replying to ronaldws:

Replying to fperrad:

I obtain the following output : ...

gdb -args parrot.exe t/library/pcre_2.pir

GNU gdb (GDB) 7.1 (gdb) r ok 1 ok 2 ok 3 not ok 4 not ok 5

I obtain : {{{

prove -v t\library\pcre.t

t\library\pcre.t .. 1..2 ok 1 - libpcre loading not ok 2 - soup to nuts # Failed test 'soup to nuts' # at t\library\pcre.t line 104. # got: 'ok 1 # ok 2 # not ok 3 # ok value seems to be error code - value is: -2 # not ok 4 # not ok 5 # ' # expected: 'ok 1 # ok 2 # ok 3 # ok 4 # ok 5 # ' # Looks like you failed 1 test of 2. }}}

After looking through these results I noticed that on the new run of test 2, step 3 failed which had not been the case with the earlier running of the test. In theory it would seem that step 3 returns an error diagnostic string but, on testing that error return string I came accross a bug in retrieving it, documented and fixed by ticket #1795. After that ticket gets accepted or it's patches are otherwise applied on your system and parrot is rebuilt, Mr. Perrad should be able to apply the latest debugging paches to /t/library/pcre.t and post the results here for further progress in tracking down why the pcre nci interface is having trouble on his system

[fperrad]

I applied all patches from TT#1795.

But the result is not constant/stable.

It could be OK, see  http://smolder.parrot.org/app/projects/report_details/140

or

ok 1
ok 2
not ok 3
not ok 4
not ok 5
not ok 6

or

ok 1
ok 2
ok 3
ok 4
not ok 5
not ok 6

or

ok 1
ok 2
ok 3
ok 4
ok 5
not ok 6

[ronaldws]

Diagnostic patch for tests

[ronaldws]

Thank you again for your time and effort on this. Please just apply one more patch, the updated pcre.t.dbg_patch, which should give us diagnostic information from the pcre library when steps 4 and 5 fail and post some output results, if you can, from test results, preferably both where 4 fails and where 4 is ok and 5 fails.

[fperrad]

A new set of output :

ok 1
ok 2
not ok 3
compile error is: \ at end of pattern
not ok 4
ok value seems to be error code - value is: -2
not ok 5
not ok 6

ok 1
ok 2
ok 3
ok 4
ok value seems to be error code - value is: -1
not ok 5
not ok 6

ok 1
ok 2
ok 3
ok 4
ok 5
not ok 6

[ronaldws]

Replying to fperrad:

A new set of output : {{{ ok 1 ok 2 not ok 3 compile error is: \ at end of pattern not ok 4 ok value seems to be error code - value is: -2 not ok 5 not ok 6 }}}

I can reproduce similar output by changing pat= 'a' to pat = 'a\\' in step 4 but step 3 still runs OK. It looks like some sort of memory corruption or a buffer overrun. I sort of wonder if running t/pmc/nci.t might also turn up some intermittent issues on your system. Again I am not sure it's a generic Win32 problem because I don't see anything like it on either my Win XP or my Vista system. Unless something else turns up this may be as far as I can take the problem for now.

Ron

[fperrad]

The branch gc_massacre seems to solve the problem (without any patch).

See  http://smolder.parrot.org/app/projects/report_details/163

[jkeenan]

I am going to mark this ticket closed because there have been no further posts in 5 months and because the gc_massacre branch no longer exists (presumably, its essentials have been merged into master).

kid51