Parrot needs to become a stable platform for HLL development. A big component of that is finding out quickly when a change breaks an HLL, and automated HLL testing will help. We also need to fix Parrot's culture of looking at our own test suite as the primary arbiter of what constitutes a safe change. Hopefully regular automated HLL testing will help ingrain that in our collective mindset.
HLL testing needs
An HLL CI environment needs to:
- test the latest version of Parrot against known-good versions of a variety of HLLs and libraries
- nqp-rx
- plumage
- rakudo
- intelligently increment the versions of HLLs
- needs to hold parrot steady so that we can be relatively certain that the upgrade didn't introduce an HLL bug
- testing old versions of HLLs that are expected to retain backwards-compatibility may be desirable
- be smart enough to know that an old HLL won't work on a current Parrot
- deprecations and support periods may come into play here
- notify #parrot when something breaks an HLL
- granularity can be configurable, but every couple hours would be sufficient. Hourly or per-commit would be ideal.
- be able to test multiple builds, i.e. different C compilers and Configure.pl flags
- at least --optimize and normal
- keep notifying #parrot every time the HLL stays broken
- it might even retry on every commit once an HLL is known to fail
- notify #parrot when an HLL stops being broken
- live somewhere where it can be reasonably modified by more than one parrot developer
- any interested and trusted developer should be able to get an admin bit
- low bus numbers lead to pain