bytecode testing framework

[allison]

A framework for directly testing bytecode examples. Somewhat similar to the PIR/PASM/C testing framework.

[Util]

In #parrotsketch,  http://irclog.perlgeek.de/parrotsketch/2009-09-22#i_1527763 , Allison asked me to put together a first draft of what I think might need to be done. I see two distinct needs that shape the issue:

1) Now that Packfile PMCs work, some tests, written in PIR, will need to create or manipulate a separate piece of bytecode, and need to run the resulting .pbc using the same parrot.exe that is running the test PIR. The .pbc never has no true equivalent .pir source on disk. For example, examples/pir/make_hello_pbc.pir creates its own generated_hello.pbc, then exercises it by load_bytecode 'generated_hello.pbc'. How should this be converted into a t/.t test? What services should the test framework provide to support this kind of testing, to reduce redundant code? Is it healthy to run that last step via load_bytecode, or should a separate invocation of ./parrot generated_hello.pbc be used? One landmine to avoid is (just) compiling make_hello_pbc.pir to .pbc; that is, conflating this with part (2), below.

2) Because IMCC can load-and-run .pir without writing bytecode to a file, a (possible?) gap exists between the codepaths of running ./parrot FOO.pir vs ./parrot -o FOO.pbc FOO.pir && ./parrot FOO.pbc. It is possible for the output of these two cases to disagree, although this would always indicate a bug. We need to be able to run any .pir test in either codepath (direct .pir, or .pir->.pbc-then-run).

How to approach? What API?

• Extend current testing framework to have a flag that means "invisibly compile this PIR to .pbc, and run the .pbc instead of the .pir"?
• ENV var to force everything that would have been run as .pir to be run as .pbc?
• ENV var to force everything that would have been run as .pir to be run and tested *twice*, once as .pir and once as .pbc?
• Magic setting in use Test::More line?
• Make a new dimension in the make testing targets? (make testc vs make testc-pbc)

.t needs to do the right thing.

Actual coding seems straightforward, except for the hierarchal aspect of exceptions, which might not be captureable.

[dukeleto]

I think both codepaths should always be run, so I would vote for each test that tests bytecode to try both paths.

[dukeleto]

In light of this talk on #parrotsketch

<Util>: TT#600  (bytecode testing framework) - only progress is a vote from dukeleto++
 for running both .pir and .pbc versions of the testing as the default. I need to 
carry it into #parrot; among other issues, running both will more than double our
current `make test` time
<allison> Util: aye, it's something that should be an option for fulltest, but not part of the default tests

I think that if both version were run on fulltest, and only one on a normal test run, that would be fine. An optional flag is also fine.

I would just like to note that getting 100% test coverage should always be more important than making our test suite slower.

[mikehh]

perhaps I have the wrong end of the stick, but is this not what testr does.

It compiles to bytecode and then tests the bytecode: .pir -> .pbc files and test .pbc

[barney]

Replying to mikehh:

perhaps I have the wrong end of the stick, but is this not what testr does. It compiles to bytecode and then tests the bytecode: .pir -> .pbc files and test .pbc

Actually 'make testr' does something a little bit different. It creates a '*.pbc' File, converts it into an executable and runs the executable.

It doesn't run the pbc-File directly.

[coke]

Replying to Util:

2) Because IMCC can load-and-run .pir without writing bytecode to a file, a (possible?) gap exists between the codepaths of running ./parrot FOO.pir vs ./parrot -o FOO.pbc FOO.pir && ./parrot FOO.pbc. It is possible for the output of these two cases to disagree, although this would always indicate a bug.

Given :immediate, I'm not sure this true.

[coke]

Bumping milestone to 2.0; This should be reviewed in light of the recent roadmap meeting.