Changes between Initial Version and Version 1 of Perl 5 Components: Test Coverage Of Configuration And Build Tools

Show
Ignore:
Timestamp:
12/10/08 06:59:33 (6 years ago)
Author:
leto
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Perl 5 Components: Test Coverage Of Configuration And Build Tools

    v1 v1  
     1From November 2006 to the present extensive efforts have been made to increase the extent to which Parrot's various test suites exercise the Perl 5 source code used during the configuration and build phases of the Parrot installation process. 
     2 
     3== Parrot Installation Phases == 
     4 * The *configuration* phase begins with `perl Configure.pl` and culminates 
     5with the creation of '''Makefiles''' and other files needed for the build 
     6phase. 
     7 
     8 * The *build* phase begins with `make` and culminates in the creation of the '''parrot''' executable. 
     9 * Once the '''parrot''' executable is available, you can proceed to the *test*  
     10phase (`make test`) and/or the *install* phase (`make install`).  For the 
     11purpose of this wiki page, we are not concerned with the test and installationphases. 
     12 
     13== Configuration and Build Tools Test Suites == 
     14A number of test suites and `make` targets have been developed for the purpose of providing unit-test-level testing of the Perl 5 source code used in the co 
     15nfiguration and build phases.  Some of these can -- and should -- be run before '''make''' is run. These include: 
     16 * `perl Configure.pl --test=configure`:  This runs the suite of tests found in '''t/configure/''' then runs '''Configure.pl''' itself. * `perl Configure.pl --test=build`:  This runs runs '''Configure.pl''', then runs the suites of tests found in '''t/postconfigure/''', '''t/tools/pmc2cutils/''', '''t/tools/ops2cutils/''' and '''t/tools/ops2pmutils/'''. 
     17 
     18 * `perl Configure.pl --test`:  This combines the operation of `--test=configure` and `--test=build`, '''i.e.''', it runs the pre-configuration tests, then  
     19'''Configure.pl''' itself, then the post-configuration and build tests. 
     20 
     21Other test suites are structured as '''make''' targets and so need Parrot's '''MAKEFILE''' to have been created before they can be run -- which implies that 
     22 `perl Configure.pl` has completed its run.  These targets include: 
     23 
     24 * `make buildtools'''tests`:  This runs the suites of tests found in '''t/tools/pmc2cutils/''', '''t/tools/ops2cutils/''' and '''t/tools/ops2pmutils/'''.   
     25These tests simulate what happens when `make` invokes three of Parrot's build tools:  '''tools/build/pmc2c.pl''', '''tools/build/ops2c.pl''' and '''tools/bu 
     26ild/ops2pm.pl'''. 
     27 
     28 * `make manifest'''tests`:  This tests the code underlying tools/dev/mk_manifest_and_skip.pl. 
     29 
     30== Coverage Analysis of Configuration and Build Tools Tests == 
     31 
     32 
     33The thoroughness of our efforts to test Parrot's configuration and build tools 
     34is largely measured by [http://en.wikipedia.org/wiki/Code_coverage coverage analysis]:  running our test 
     35suites with coverage options turned on.  For Parrot's Perl 5 components, that 
     36means running our tests while having Paul Johnson's CPAN module 
     37[http://search.cpan.org/dist/Devel-Cover/ Devel::Cover] turned on.  Here is an 
     38example of a shell script which was used at various points to perform coverage 
     39analysis on the '''reconfigure/''' branch in the Parrot Subversion repository: 
     40 
     41{{{ 
     42COVERAGEDIR=/webviewable_directory/parrot/coverage/configure-build 
     43RECONFDIR=/home/user/work/reconf 
     44coverconf () 
     45{ 
     46cd $RECONFDIR && \ 
     47cover -delete $COVERAGEDIR && \ 
     48PERL5OPT=-MDevel::Cover=-db,$COVERAGEDIR/ \ 
     49perl $RECONFDIR/Configure.pl --test --configure'''trace "$@"  && \ 
     50cover $COVERAGEDIR \ 
     51-ignore_re 'Configure\.pl' \ 
     52-ignore_re '^t\/' \ 
     53-ignore_re '^\/usr\/local\/bin' \ 
     54-ignore_re '^lib\/(?!Parrot)' && \ 
     55chmod 0755 $COVERAGEDIR/ 
     56} 
     57}}} 
     58 
     59Note that by running `perl Configure.pl --test` with Devel::Cover turned on, we exercise the code within and underlying '''Configure.pl''' by running the test suites and that program itself. 
     60 
     61== What Remains to Be Done? == 
     62 
     63To come 
     64 
     65== F.A.Q. == 
     66 
     67 * What is the relationship between the configuration and build tools tests and the test suite run by `make test`?* 
     68 
     69The configuration and build tools tests have no inherent relationship to the tests run by `make test`.  Neither of those suites assumes that the '''parrot''' executable has been created, and nothing in those suites tests whether the '''parrot''' executable performs according to its specifications.  So there's no reason to wait until '''parrot''' has been created -- '''i.e.,''' until after `make` has been created to run these tests.  Both suites are more logically run '''before''' '''make''' has been invoked. 
     70 
     71The build tools tests, for example, test whether the build tools will be able to perform to '''their''' specification '''when''' '''make''' invokes them.  If '''make''' has already completed successfully, your build tools have, by definition, succeeded, so further testing of them during `make test` is superfluous.  You want to know whether your build tools will work before `make` invokes them.  And, indeed, the build tools test are set up to test whether `make` has been run or not -- and they fail if `make` has been run.  The build tools tests ought to be run by Parrot developers hacking on the build phase, but they would have little relevance to, say, Parrot developers working on language implementations. 
     72 
     73The pre-configuration tests, as the term implies, are ideally meant to be run '''before''' `perl Configure.pl`.   They test whether the code underlying '''Configure.pl''' is up to the task which '''Configure.pl''' sets out for it. 
     74 
     75The post-configuration tests are something of a special case.  Certain Parrot developers have need to re-run individual configuration steps and regenerate a '''MAKEFILE''' once configuration as a whole has run and the various '''MAKEFILEs''' have been created.  The post-configuration tests are, therefore, ideally meant to be run 'post-'''Configure.pl''', pre-'''make'''.' 
     76 
     77The pre-configuration tests and the post-configuration tests are run as part of `make test`, but that's largely for the historical reason that a few tests of the configuration tools were run as part of the `make test` suite before this intensive effort at testing began. 
     78 
     79