Ticket #432 (closed bug: fixed)

Opened 13 years ago

Last modified 13 years ago

Running `make fulltest` does not test fully.

Reported by: Util Owned by: Util
Priority: major Milestone:
Component: configure Version: trunk
Severity: medium Keywords: fulltest make makefile fulltest_all
Cc: allison, coke Language:
Patch status: Platform: all

Description

Target "fulltest" behaves in a less-than-useful fashion; if any of the groups of tests fail, then none of the later groups will run at all.
For example, if the "doc_tests" group fails, then "manifest_tests" will not run.

Since all the variations of the runcore tests (which rarely *all* pass) are ordered first, testers who rely on the completeness of target "fulltest" may have *never* run the later-ordered meta-quality tests.

Target "fulltest" should ignore errors in its sub-targets. I think that fulltest was *intended* to do just that, but the single dash before the single "make" command is a mis-coding of the concept. A separate "make" for each sub-target is needed, each preceded with its own error-ignoring dash.

Change History

Changed 13 years ago by Util

  • status changed from new to closed
  • resolution set to fixed

Fixed in r37269.

Changed 13 years ago by coke

  • status changed from closed to reopened
  • resolution fixed deleted

Changed 13 years ago by coke

'make fulltest' is supposed to fail if any of the subtests fail, so we actually know that they are failing.

If one fulltest runs 7 different invocations of t/harness, and the first one fails, the failure is now lost in the output, and only the last invocation is visible to the user; and that one may have passed 100%, giving the person running the tests the wrong impression.

There are two ways to resolve this issue: get 'make fulltest' passing all tests, or combine all the various sub-targets into a single harness (This was hard at one point. not sure if it still is.)

I would revert this change, unless there was some discussion that occurred outside of this ticket.

Changed 13 years ago by Util

Reverted in r37271.
I will re-submit it as a separate target later today.

Changed 13 years ago by Util

  • status changed from reopened to closed
  • keywords fulltest_all added
  • resolution set to fixed
  • cc coke added

Resolved in r37272, by adding new target fulltest_all, which runs all the test groups regardless of failure.

The old target fulltest retains its old meaning, stopping on the first test group failure.

Note: See TracTickets for help on using tickets.