Version 1 (modified by leto, 6 years ago)


From November 2006 to the present extensive efforts have been made to increase the extent to which Parrot's various test suites exercise the Perl 5 source code used during the configuration and build phases of the Parrot installation process.

Parrot Installation Phases

  • The *configuration* phase begins with perl and culminates

with the creation of Makefiles and other files needed for the build phase.

  • The *build* phase begins with make and culminates in the creation of the parrot executable.
  • Once the parrot executable is available, you can proceed to the *test*

phase (make test) and/or the *install* phase (make install). For the purpose of this wiki page, we are not concerned with the test and installationphases.

Configuration and Build Tools Test Suites

A number of test suites and make targets have been developed for the purpose of providing unit-test-level testing of the Perl 5 source code used in the co nfiguration and build phases. Some of these can -- and should -- be run before make is run. These include:

  • perl --test=configure: This runs the suite of tests found in t/configure/ then runs itself. * perl --test=build: This runs runs, then runs the suites of tests found in t/postconfigure/, t/tools/pmc2cutils/, t/tools/ops2cutils/ and t/tools/ops2pmutils/.
  • perl --test: This combines the operation of --test=configure and --test=build, i.e., it runs the pre-configuration tests, then itself, then the post-configuration and build tests.

Other test suites are structured as make targets and so need Parrot's MAKEFILE to have been created before they can be run -- which implies that

perl has completed its run. These targets include:

  • make buildtools'''tests: This runs the suites of tests found in t/tools/pmc2cutils/, t/tools/ops2cutils/ and t/tools/ops2pmutils/.

These tests simulate what happens when make invokes three of Parrot's build tools: tools/build/, tools/build/ and tools/bu ild/

  • make manifest'''tests: This tests the code underlying tools/dev/

Coverage Analysis of Configuration and Build Tools Tests

The thoroughness of our efforts to test Parrot's configuration and build tools is largely measured by  coverage analysis: running our test suites with coverage options turned on. For Parrot's Perl 5 components, that means running our tests while having Paul Johnson's CPAN module  Devel::Cover turned on. Here is an example of a shell script which was used at various points to perform coverage analysis on the reconfigure/ branch in the Parrot Subversion repository:

coverconf ()
cd $RECONFDIR && \
cover -delete $COVERAGEDIR && \
PERL5OPT=-MDevel::Cover=-db,$COVERAGEDIR/ \
perl $RECONFDIR/ --test --configure'''trace "$@"  && \
-ignore_re 'Configure\.pl' \
-ignore_re '^t\/' \
-ignore_re '^\/usr\/local\/bin' \
-ignore_re '^lib\/(?!Parrot)' && \
chmod 0755 $COVERAGEDIR/

Note that by running perl --test with Devel::Cover turned on, we exercise the code within and underlying by running the test suites and that program itself.

What Remains to Be Done?

To come


  • What is the relationship between the configuration and build tools tests and the test suite run by make test?*

The configuration and build tools tests have no inherent relationship to the tests run by make test. Neither of those suites assumes that the parrot executable has been created, and nothing in those suites tests whether the parrot executable performs according to its specifications. So there's no reason to wait until parrot has been created -- i.e., until after make has been created to run these tests. Both suites are more logically run before make has been invoked.

The build tools tests, for example, test whether the build tools will be able to perform to their specification when make invokes them. If make has already completed successfully, your build tools have, by definition, succeeded, so further testing of them during make test is superfluous. You want to know whether your build tools will work before make invokes them. And, indeed, the build tools test are set up to test whether make has been run or not -- and they fail if make has been run. The build tools tests ought to be run by Parrot developers hacking on the build phase, but they would have little relevance to, say, Parrot developers working on language implementations.

The pre-configuration tests, as the term implies, are ideally meant to be run before perl They test whether the code underlying is up to the task which sets out for it.

The post-configuration tests are something of a special case. Certain Parrot developers have need to re-run individual configuration steps and regenerate a MAKEFILE once configuration as a whole has run and the various MAKEFILEs have been created. The post-configuration tests are, therefore, ideally meant to be run ', pre-make.'

The pre-configuration tests and the post-configuration tests are run as part of make test, but that's largely for the historical reason that a few tests of the configuration tools were run as part of the make test suite before this intensive effort at testing began.