Ticket #1941 (new cage)

Opened 4 years ago

Last modified 4 years ago

make warnings_tests: Correct tests which run with warnings

Reported by: jkeenan Owned by:
Priority: minor Milestone:
Component: testing Version: 2.11.0
Severity: medium Keywords:
Cc: cotto, mikehh Language:
Patch status: Platform:

Description

In the course of work on TT #1922 this evening, I had occasion to run make warnings_tests for the first time. This target will FAIL test files which would otherwise PASS if they emit warnings. We should fix all tests such that they run warnings-free.

Here are the files which failed due to warnings:

Test Summary Report
-------------------
t/compilers/imcc/syn/const.t      (Wstat: 1280 Tests: 35 Failed: 5)
  Failed tests:  15-18, 21
  Non-zero exit status: 5
t/compilers/imcc/syn/errors.t     (Wstat: 1280 Tests: 7 Failed: 5)
  Failed tests:  2-5, 7
  Non-zero exit status: 5
t/compilers/imcc/syn/macro.t      (Wstat: 512 Tests: 42 Failed: 2)
  Failed tests:  31-32
  Non-zero exit status: 2
t/compilers/imcc/syn/pasm.t       (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
t/compilers/imcc/syn/regressions.t (Wstat: 256 Tests: 21 Failed: 1)
  Failed test:  15
  Non-zero exit status: 1
t/op/basic.t                      (Wstat: 256 Tests: 23 Failed: 1)
  Failed test:  23
  Non-zero exit status: 1
t/pmc/exception-old.t             (Wstat: 256 Tests: 22 Failed: 1)
  Failed test:  16
  Non-zero exit status: 1
t/pmc/filehandle.t                (Wstat: 512 Tests: 27 Failed: 2)
  Failed tests:  3, 21
  Non-zero exit status: 2
t/pmc/nci.t                       (Wstat: 1280 Tests: 73 Failed: 5)
  Failed tests:  1-3, 60-61
  Non-zero exit status: 5
t/pmc/stringhandle.t              (Wstat: 256 Tests: 25 Failed: 1)
  Failed test:  9
  Non-zero exit status: 1
t/pmc/sub.t                       (Wstat: 768 Tests: 78 Failed: 3)
  Failed tests:  31-32, 44
  Non-zero exit status: 3
Files=205, Tests=7028, 66 wallclock secs ( 0.87 usr  0.39 sys +
 37.84 cusr 10.37 csys = 49.47 CPU)
Result: FAIL
make: *** [warnings_tests] Error 1

Attachments

t_compilers_imcc_syn_errors.diff Download (487 bytes) - added by jkeenan 4 years ago.
Have this file pass 'make warnings_tests'
t_pmc_sub_make_warnings_tests_fail.txt Download (0.9 KB) - added by jkeenan 4 years ago.
output from 'make warnings_tests' for this file
t_pmc_nci_make_warnings_tests_fail.txt Download (2.2 KB) - added by jkeenan 4 years ago.
output from 'make warnings_tests' for this file

Change History

  Changed 4 years ago by jkeenan

  • summary changed from make warnings_tests: Eliminate tests which run with warnings to make warnings_tests: Correct tests which run with warnings

  Changed 4 years ago by jkeenan

  • cc cotto, mikehh added

I spot-checked the individual test files listed above. It appears that in all, or almost all, cases, the tests that are "running with warnings" are ... tests that are testing warnings!

Which suggests that make warnings_tests is not a very useful operation. Or, at least not until we have for PIR tests something that is the equivalent of IO::CaptureOutput::capture() which we use in Perl 5 tests.

Since I doubt that many people have ever used make warnings_tests and since it would take a lot of work to make it do something useful, my recommendation is that we drop it.

Thoughts?

kid51

follow-up: ↓ 4   Changed 4 years ago by cotto

Can the getstderr and setstderr ops be used for this?

in reply to: ↑ 3   Changed 4 years ago by bacek

Replying to cotto:

Can the getstderr and setstderr ops be used for this?

Yes, it's possible. I used them in pirate test suite, afair.

Changed 4 years ago by jkeenan

Have this file pass 'make warnings_tests'

Changed 4 years ago by jkeenan

output from 'make warnings_tests' for this file

Changed 4 years ago by jkeenan

output from 'make warnings_tests' for this file

follow-up: ↓ 6   Changed 4 years ago by jkeenan

1. As of today (commit d76e2cc1) the number of tests failing under make warnings_tests is much smaller.

Test Summary Report
-------------------
t/compilers/imcc/syn/errors.t     (Wstat: 256 Tests: 8 Failed: 1)
  Failed test:  7
  Non-zero exit status: 1
t/pmc/nci.t                       (Wstat: 1280 Tests: 73 Failed: 5)
  Failed tests:  1-3, 60-61
  Non-zero exit status: 5
t/pmc/sub.t                       (Wstat: 768 Tests: 78 Failed: 3)
  Failed tests:  31-32, 44
  Non-zero exit status: 3
Files=208, Tests=7094, 71 wallclock secs ( 0.99 usr  0.42 sys + 
  40.23 cusr  9.89 csys = 51.53 CPU)
Result: FAIL
make: *** [warnings_tests] Error 1

2. I am attaching a patch which eliminates the failure in t/compilers/imcc/syn/errors.t. Please review.

3. I am attaching two other patches which display the sections of the output of 'make warnings_tests' relevant to t/pmc/nci.t and t/pmc/sub.t. I wonder: Are the tests written in such a way as to anticipate the possibility of warnings output or other output to STDERR?

in reply to: ↑ 5   Changed 4 years ago by jkeenan

Replying to jkeenan:

2. I am attaching a patch which eliminates the failure in t/compilers/imcc/syn/errors.t. Please review.

mikehh applied this particular patch in b058323a on Feb 12. So we're down to the failures in t/pmc/nci.t and t/pmc/sub.t.

Note: See TracTickets for help on using tickets.