Ticket #432 (closed bug: fixed)

Opened 6 years ago

Last modified 6 years ago

Running `make fulltest` does not test fully.

Reported by: Util Owned by: Util
Priority: major Milestone:
Component: configure Version: trunk
Severity: medium Keywords: fulltest make makefile fulltest_all
Cc: allison, coke Language:
Patch status: Platform: all

Description

Target "fulltest" behaves in a less-than-useful fashion; if any of the groups of tests fail, then none of the later groups will run at all.
For example, if the "doc_tests" group fails, then "manifest_tests" will not run.

Since all the variations of the runcore tests (which rarely *all* pass) are ordered first, testers who rely on the completeness of target "fulltest" may have *never* run the later-ordered meta-quality tests.

Target "fulltest" should ignore errors in its sub-targets. I think that fulltest was *intended* to do just that, but the single dash before the single "make" command is a mis-coding of the concept. A separate "make" for each sub-target is needed, each preceded with its own error-ignoring dash.

Change History

Changed 6 years ago by Util

  • status changed from new to closed
  • resolution set to fixed

Fixed in r37269.

Changed 6 years ago by coke

  • status changed from closed to reopened
  • resolution fixed deleted

Changed 6 years ago by coke

'make fulltest' is supposed to fail if any of the subtests fail, so we actually know that they are failing.

If one fulltest runs 7 different invocations of t/harness, and the first one fails, the failure is now lost in the output, and only the last invocation is visible to the user; and that one may have passed 100%, giving the person running the tests the wrong impression.

There are two ways to resolve this issue: get 'make fulltest' passing all tests, or combine all the various sub-targets into a single harness (This was hard at one point. not sure if it still is.)

I would revert this change, unless there was some discussion that occurred outside of this ticket.

Changed 6 years ago by Util

Reverted in r37271.
I will re-submit it as a separate target later today.

Changed 6 years ago by Util

  • status changed from reopened to closed
  • keywords fulltest_all added
  • resolution set to fixed
  • cc coke added

Resolved in r37272, by adding new target fulltest_all, which runs all the test groups regardless of failure.

The old target fulltest retains its old meaning, stopping on the first test group failure.

Note: See TracTickets for help on using tickets.